Free jazz was a major movement in music in the early- to mid-1960s. Its influence can still be felt in these more contemporary tracks you posted.
The earliest free jazz recordings were probably the Ornette Coleman Atlantics and the earliest work by Cecil Taylor. These came out in the late 50s and early 60s. Most of those recordings featured a much freer tonality but still a fairly conventional timekeeping role from the rhythm section.
But the freedom from keeping strict tempo really gained steam when Coltrane started to employ it with his quartet. By the time Trane brought Rashied Ali into the group, it was on. Artists like Albert Ayler, with the great Sunny Murray on drums, explored the style more thoroughly. Other artists who started in the more traditional post-bop forms but explored free jazz were Eric Dolphy, Charles Mingus, Archie Shepp and Sun Ra. Coleman and Taylor continued to get farther and farther "out" as well.
There was concurrently a movement in Europe where musicians coming from a different musical background explored collective improvisation. The FMP label was where a lot of it happened, and important musicians like Peter Brotzmann and Evan Parker came up with their own take, independent of the American free jazzers.
As far as why it came to be, well, I think it was just a logical progression, an attempt to take the music to the next level. Jazz was evolving so quickly during that time, getting freer and freer all the time, opening up more avenues to the improviser. Coltrane, in particular, seemed to be searching for something almost spiritual, like trying to discover some new language or reality. The way bebop startled the swing generation, free jazz shocked the boppers.
We had a nice thread going on this forum with free jazz recs somewhere. It might even be on the early pages of this thread.