Here is a video of an actual successful speed record attempt. The gliders pull high Gs and use composite construction to take the load. I wonder what the G load would be on a full-size, piloted glider doing the same thing on a larger scale? I doubt there are any built that could take it, assuming first that the pilot could stand it.
Here’s the news (at least to me), 15 years late. Turns out hobbyists have done something called dynamic soaring using radio-controlled (RC) gliders since the late 1990s. The current record (unofficial) stands at 391 mph. You can learn more about it here.