One thing that’s been keeping us very busy at CodePen is moving our main API. We decided on GraphQL long ago and it’s served us pretty well. We originally built it in Ruby on Rails alongside a lot of the rest of our app. But while Rails served us well, we’ve been moving off of it. We like our React architecture and we’re better served leaning into that, with frameworks like Next, than staying on Rails. We proved out this combination of technologies for ourselves, building a whole set of admin tools with it. Now we’re ready to keep that train going as we build out more of CodePen with the same stack. But removing Rails means moving off of our Rails-based GraphQL implementation. This means re-writing that API in Go, another bit of tech we’ve had a lot of luck with.
Turns out that re-writing an API is more time-consuming than writing it to begin with, especially as we need to run them side-by-side and behave identically. No refactoring allowed! Unless of course we want to refactor it on both sides and take even more time.
Dee joined me this week in talking about all this. It’s a huge job! But we’ve been doing well at it, building our own tooling, doing lots of testing, and ultimately proving that it works by releasing it in small areas on the production site. It’s all working out how we hoped it would: fast, cheap, and easier to reason about.
Equinix Metal’s Startup Partner ProgramSponsor:
Equinix Metal’s Startup Partner Program helps early stage companies level up. Their experts work with startups like Koord and INVISV to build their competitive edge with infrastructure. Equinix Metal provides real time guidance and support to help startups grow faster. With up to $100,000 in infrastructure credit, access to Equinix’s global ecosystem of over 10,000 customers and 1,800 networks, they might just be what you need to take your startup global.
Visit metal.equinix.com/startups to take your startup to the next level.
[Radio channel adjustment]
Announcer: Today, on CodePen Radio.
Chris Coyier: Hey, everybody. CodePen Radio #389. I want to talk about something near and dear to our hearts, which is some of the work we've been doing internally that's just been [laughter] taking forever but feels satisfying to do. Because it's not particularly user-facing, it's not worth a blog post every week or anything (about it). But it's a big deal to us, and that's why we have this podcast because CodePen Radio is all about talking about what it's like running a software company today - and it has been for a lot of years.
I have with me Dee. Dee, what's up?
Dee Vazquez: Hey, Chris! How are you? It's been a hot minute.
Chris: I know!
Chris: It has been a hot minute. We were trying to figure out when the last time you've been on the show. And it's not because - I don't know - you could be on the show every week if you ask me.
Chris: We work together every day.
Chris: But life is life, and I think we were talking about finance and stuff the last time you were on the show, which is funny because it's like you totally do finance for CodePen.
Chris: But how big of a chunk of that is your actual work at CodePen, I don't know.
Dee: You'd be surprised.
Chris: Ten percent or something?
Dee: The last couple of weeks have been hot and heavy in the finance world, and the year budgeting and stuff. But I think it was more of the two babies that's caused this, like, drought of me being on CodePen Radio.
Chris: Yeah. Yeah, right.
Dee: I was looking, and I think I was here September 21st, so literally a month before we had our twins.
Chris: Oh, so it's been over a year since you've been on the show? Oh, my gosh.
Dee: Yeah, I know.
Chris: Well... Yeah, well, you've been hard at work being a mom and hard at work at CodePen itself, being a developer here.
We're going to do dev stuff this time, big-time dev stuff, which you are super involved in this. Most of us were, at CodePen.
Chris: This, as you can see from the show title, is about creating an API, GraphQL API actually, in Go. Really, it's rewriting it, which is funny. I should say at the top of the show, I think - and I wonder if you agree with me or not - that rewriting this API has been much more work than writing the API.
Dee: Yeah. It's funny that you say that. When I hear rewriting, I'm like, "Oh, brand new logic." No, not at all. It's actually going through our massive Rails codebase and porting it exactly.
The process was: Look at the Rails code. Figure out how to do that in Go. Put it in Go.
You would think, since the logics are even written, that it would be simple.
Dee: It's not like new logic. It's literally just moving it, porting it.
Chris: I know. It's like - I don't know - make the intern do it or something. Whoa!
Chris: A) Absolutely not.
Chris: It's very tricky because every bone in your body -- yours, I'm sure, especially -- wants to rewrite things to be better, like, "Oh, why did we name it that? That's a stupid name. I'm going to name it the good, new way."
No. You're not, actually, because if you are refactoring an API as you're moving it, you're going to have a hell of a time because, as we refactor it, we're not just turning off the old API. That is not how this is going to go.
I know we've jumped ahead a little bit, so we're going to need to provide a little bit more foundation. But I wanted to do some juicy stuff right at the top of the podcast for people to sink their teeth into.
Chris: Yeah. The point is we're going to have two APIs that kind of coexist and do the same things. If you have simultaneous APIs, they really need to do exactly the same thing. And that meticulousness, the fact that these APIs need to behave identically in two different languages - woo! Tricky, tricky, tricky, tricky.
Dee: Right. Yeah, and the focus wasn't to move over our logic and improve it. The focus was more to serve a future project or an in-progress project, kind of setting the stage to build on top of our existing stack. And so, it was really more about language, speed, costs, and then very little about building a better version of the actual logic.
Chris: Right. We can make it better over time once it's done because there is a goal at the very, very, very end of this that we do turn off the old API. I don't want to support two APIs. It's just that's silly. I think we're grown up enough to think that we can just rewrite the whole API and turn off the old one and turn on the new one and just call it.
The risk there is nuts. Why would you do that? [Laughter]
Dee: Yeah. That was part of the decision-making, as we were trying to figure out whether we should keep parts of the Rails API up and running and then just do this new project that we're working on in the new Go-based API. There was lots of discussions. There was an API inventory, like, would it be faster just to do the new feature set in the new API and keep the old API running?
Chris: Keep the old one around. Yeah, yeah.
Dee: For developer sanity, managing two code bases in two different languages is not something that we want to support because we're a super small team.
Dee: Our advantage, as a small team, is to be able to work quickly. Having to keep the Rails API in your head and the Go API in your head is not fun. As a team, all of us have basically moved over to Go, the language, from Ruby on Rails. Hopefully, within a year or so, the majority of our team's time will be spent in Go and not in Rails, which is interesting because we've been working in Rails since inception, since 2011. We had much more expertise in Ruby on Rails. But now we're building up our expertise in Go.
Chris: Not any more.
Chris: Yeah, so we need to establish the why here. Why would this team? And you answered it to some degree. It's because our team has more expertise in Go, so that's why we would make the API in Go.
There are other reasons that are probably obvious to anyone who has listened to our podcast or knows anything about Go. It's really fast, so that's a reason to do it right there.
You said in your notes here, we've talked about Go already in a bunch of other episodes, so we'll just link those up. Go is the right answer for us for a variety of reasons. That's kind of why we're doing it.
But there's a little bit more history to establish that I think is kind of fun that I think people that have been running software apps might get a kick out of is seeing how the app evolved and why we're like, "Okay, let's do this." You know?
Chris: Way old school, Ruby on Rails - whatever. No GraphQL in sight. No React in sight. Nothing fancy at all. jQuery, you know?
Then we add React to the stack. I think, when we added React, we did not add GraphQL. Weird.
Dee: No, that's right. We did not. I think React was added back in like early 2016 or so, and then maybe in, like, 2017, we added GraphQL and Apollo Client.
Dee: We moved from a traditional REST API to GraphQL, which the entire--
Chris: So weird to me. I wonder what it was like. How were we mutating data from React without GraphQL? That was a weird phase. It didn't last that long, but we did it.
Then we added GraphQL, and GraphQL was a good choice for us. We still have that. Most of CodePen is GraphQL powered, yadda-yadda. You know? Most of our app is React, but not all.
We even call it... [Laughter] This feels weird to say publicly. We call it "strangling." I think Alex likes to use that word.
Dee: [Laughter] Mm-hmm.
Chris: Strangling Rails. Let's get rid of it. Ack! Not because it hasn't been good to us, but to small up our technology set.
Chris: And focus it around stuff like that.
Dee: CodePen was built in Ruby and Sinatra was the first framework that you guys picked back in 2011.
Dee: Then you moved on to Rails because it's so fast and easy to build an app from scratch. Now that we're almost at nine million users and we process millions and millions of requests to bundle up code, Ruby on Rails, we've been feeling how slow it is. Not just on the user-facing side, but even deploying one of our Rails services, whether it's our internal CI system or switching out the Rails servers when we release a new bit of code.
Chris: Good point. Yeah.
Dee: It would take 10 minutes, 20 minutes sometimes, and we'd be like, "What is going on? Why is this taking so long?"
Chris: [Laughter] Yeah. Crucially, it is slow for users too. It's slow in every way it can be slow: development and for users. Not terribly for users, but certainly the parts of our app that are totally modernized are literally faster for users too, which is really what matters, especially the Go GraphQL stuff. Ooh, that's smokin'.
Dee: Yeah. Then industry-wide, whoever is handling code processing, you're seeing them move to much faster languages. We looked at esbuild -- it's open-source -- as inspiration. It's written in Go. Vercel just released Turbopack, which is Webpack written in Rust instead of Node.js, and so the super-speedy languages du jour right now are Go and Rust.
I remember seeing Rust books in the bathroom. I'm like, "Please, no. Let's switch to Go and stay with Go. I don't think I want to change CodePen yet again." But Rust and Go are just what the industry is moving to.
Chris: No. Yeah, right. Bun is powered by Zig, this other. Yeah, these next-gen backend languages. I think we're probably pretty good with Go. Our expertise is built there, and it is also extremely fast. So, I think we're pretty happy there.
We have Rails. We still have React and Apollo on the front end, right? But Rails is serving routes and stuff. I know that's weird to wrap your head around these days, but that's how it works. It's not that bad.
It's not like the whole CodePen isn't a Next.js app that's server-side routed and stuff... or client-side routed. We're still server-side routed, believe it or not. Anyway, we're trying to move to a more modern future, but our whole API is written literally in Ruby. All the resolvers and stuff for the APIs are in Ruby.
That was kind of fine for a long time. Actually, kind of good. It's really well-tested. All the robust, our most robust API is in Ruby. That's exactly what we're rewriting.
We're like, "What if we had this running on Go servers instead?" That's obviously what we've been doing. But we decided to have a particular approach to it that wasn't just "build it and ship it," right? We had a little bit more of a nuanced plan to making this thing real.
Dee: Yeah. We started with an internally-facing version of the API because we've made the mistake of adopting new technology while trying to solve a new challenging product problem. I think it was when we switched over to React and Redux. We were working on projects as well, and it was really hard for the entire team to pick up React and then Redux and then eventually move over to Apollo Client and also be building out the projects editor.
From that experience, we learned to tackle one problem at a time. We started our foray into the next version of CodePen by getting experience with our new toolset in production. We've re-wrote our internally facing admin tools as a Next.js app with Go APIs. That took a while too.
I think that's where Chris was expressing, like, "This has been taking forever. Why is it taking so long?" It's just because it's a brand new toolset that you're building or porting over a bunch of complicated code that' been built over ten years into the new toolset.
We started off by making sure we were solving one problem at a time by building out (we call it) CP Admin. Everything here is prefixed with CP.
Chris: [Laughter] Yeah. Exactly. Just because, what if we hate it? We'd like to know that before we ship it and have to live with it publicly for so long.
We decided we were going to test two new technologies.
Chris: Next, for the React App, which is pretty satisfying to work in. I've got to say a big fan of Next.
Like you mentioned, they just dropped Turbopack, I guess it is, the new Webpack.
Chris: Which was at the same announcement as Next 13. We already have a PR out for Next 13, because why not? It's not using Turbopack yet because I think... We'll, they announced it. They put it behind a special flag because who knows what that's going to break. That's a pretty big change to switch your entire bundler.
Although, they just did it in Next 13 to SWC, and now it's... You know what I mean? We've already lived through one bundler or processor change, so maybe it won't be so bad. I don't know.
Anyway, we did it. We shipped a Go-based GraphQL API that is only internally facing and paired it with Next. Build the admin app, and our admin app, frankly, rules.
Chris: It's really nice.
Chris: It's a pleasure to work on. The API is super nice. It's a very snappy, pleasurable thing to work on. So, mission accomplished.
Dee: And it was low pressure.
Dee: It was low pressure because it's not user-facing. It gave us time to learn the tools. And it didn't have the same security implications because it was only going to be our own developers using it and not being accessed by the outside world. It also allowed us to build out systems that were improvements on the Ruby on Rails system.
For example, integration tests. This was something that we learned when we added GraphQL and Apollo Client into our codebase. We built out request specs from in Rails where we would test our GraphQL endpoint. We would throw in the GraphQL code into Rails and then run a request back.
It'd be like we were running server-side queries within Rails, which was great. It was an awesome way to test the entire system and make sure you were getting the proper response. But then it always struck me as super strange that our GraphQL code was being rewritten and duplicated on the server side. It would have been really nice just to reuse the existing client-side GraphQL query and things like that.
One of the new systems that we built (while we were working on CP Admin) was an integration testing system where we're calling the GraphQL endpoint from the client and reusing our client-side code. That took a lot of work and a lot of iterations, but it's also been great because we're really big believers in incorporating our data model, threading it through every piece of the entire request, so client-side and backend. It takes a lot of work for the data model to be communicated and maintained consistently between the two sides, and these integration tests really help to maintain those conceptual models between the different developers that might be working on either side of the codebase. I thought that was super awesome.
Chris: Very cool.
[Guitar music starts]
Chris: Equinix Metal's Startup Partner Program helps early-stage companies level up. Their experts work with startups like Koord and INVISV to build their competitive edge with infrastructure.
Equinix Metal provides real-time guidance and support to help startups grow faster with up to $100,000 in infrastructure credit, access to Equinix's global ecosystem of over 10,000 customers and 1,800 networks. They might just be what you need to take your startup global.
Visit metal.equinix.com/startups to take your startup to the next level.
[Guitar music ends]
Chris: Mission accomplished with that. Then we decide we're so happy with this stack, let's build more of CodePen in it and keep going. We're going to turn off, eventually, the Rail's based GraphQL API and just replace it with a Go one. That means replacing all of our API, and all of our API is a pretty frickin' daunting task.
Chris: We decide. Somebody has this brilliant idea [laughter] that instead of shipping 100% of the API, let's pick one route on CodePen. Right? That one route will just build just enough of the API to serve that one route and, crucially, will ship it all the way to production.
Right? Isn't that what we talked about?
Dee: Yeah, we wanted a user-facing. We wanted to ship all this work that had just been internally facing because--
Dee: As developers, you want to see your work be put out to users and to get feedback from users.
Dee: It doesn't almost feel real until you get that user feedback, so we'd been doing all this work behind the scenes without seeing the benefits trickle out to users. And so, we decided to pick a page from our existing app that we could move to the next API quickly.
Dee: Hopefully, just a couple of weeks of work, something siloed that didn't touch big portions of the app. And so, we were like, "Oh, the About page. That's got to be low risk," and it turned out not so much. Not so much because, when you're a logged-in user, you keep state all throughout CodePen in your sidebar and your user menu, and that shows up on the About page as well.
And so, what we'd been hoping would be a short sprint, just a couple of weeks, ended up taking months of work where we touched every one of our core data models: users, collections, Pens, projects.
Chris: Yeah. We could have cut it off at any point, but we kept going, like, "Well, you know, we've got to do it anyway, so just put it in." You know?
Then it turned out, yeah, all you can do is open up your pins on CodePen, and just that opened up this insane... Like Dee said, we had to touch almost every one -- or maybe every one -- of our core data models.
Dee: Yeah. Yep. Yep.
Chris: Just to make it work. Our little idea was fun to pick a little less than all of our API, but it turns out we touched a ton to do it. Whatever. It's done now, and it's out, so I guess if you're listening to this right as it goes out, it may be true that the only page on CodePen that uses the GraphQL API is codepen.io/about. But darn if it doesn't do a good job of doing that.
Dee: Yeah. it's almost 2.5 times faster than the Rails GraphQL API.
Chris: Oh, is that the data? That's great!
Dee: Yeah. Yeah, it's--
Chris: Oh, I love that.
Dee: And we did little to no caching or performance improvements on the Go side. That's all Go, the language itself, versus Rails.
Chris: Yeah, it's just Go. That's great.
Dee: Yeah. So, in the end, worth it. We got something--
Chris: Not to mention it's running on some little baby server at like 10% memory usage or something. Right?
Dee: Yeah. Yeah, yeah. I mean it's only like a single -- I guess it's just a single page, but I think it's like nine large Ruby on Rails servers is what we have, and then the Go API is running like three medium-sized servers.
Dee: The CPU utilization on the Rails side is like 53%, and on the Go side it's like 7%, which is shocking.
Chris: I know it's just this one page, but we have lots of evidence that anything that we've ported to Go is just outstandingly faster.
Dee: Blazing fast.
Chris: And cheaper. There's a cost thing that goes with it. Pretty great. Yeah. Oh, gosh, look at all of these charts you have in here. Good job.
Dee: Thanks. I try to bring numbers wherever I can.
Chris: Yeah, that's wonderful. But what was it like to do this work? I think that's worth talking about a little bit. How do we bite off a piece of it and do it?
Dee: Yeah, I think you're a great person to ask that question, Chris, because at a certain point you were like, "I'm just going to learn Go, too, guys."
We've had several people on the team learn new technologies. It's kind of what we do at CodePen. We stay on top of the industry. People have said it's the reason they love working at CodePen because they get to learn so much of development. But you also got your hands in there, rolled up your sleeves, and learned Go.
Chris: Yeah, well, you did too. It was all in your footsteps, really. But yeah, it was expressed internally from you and Alex, really, that there's no reason anybody here couldn't write Go code. It's not that weird.
Chris: Yeah, and I really think that's true now because I was like, "Okay, I'll do a tour of Go, and I'll do Go by example. And I'll try to ship a couple of Lambdas."
Netlify, for example, makes it easy to ship a Go-based Lambda, so you can have some experience with what is it like to get a piece of Go into a production-like environment. That's basically all I did and then started doing it at CodePen. I was like, "Oh, yeah. This is fine." [Laughter]
Chris: It's really not a big deal. You know? It's just a language.
It's typed. That was the weirdest thing for me because, wow, does it bitch about the types a lot.
Chris: My whole day today has been, "Oh..." And it makes you care about types too.
Chris: Not just be mad about it, but that's one of the things I was looking at today is, in our database, there's a little bit of JSON code here and there. Rails is so loosey-goosey about that that there are a couple of places where sometimes it's an int and sometimes it's a string (in the JSON in our database), which is able to sneak through because databases are typed too. Like a column has to say what type it is.
Dee: Yeah, and that's a really good point. Going from Ruby on Rails, which it does so much for you under the covers and it can be a very magical feeling, and that's great because it lets you move fast and quickly, which is what CodePen needed when you guys first started it. But now the safety of a strongly typed language is what we're moving towards, both on the back-end, and we're even doing some forays into TypeScript right now, which is probably going to be a lot easier now, Chris, for you that you've been working with Go itself.
I think one of the learnings and turning points where everyone could contribute to this API was the tooling that we built that we brought from the Rails world. We internally created this DB Gen tooling, which is very similar to how you create Rails migrations and then build out some of the queries because, on the Rails side, we have a lot of that done for us whereas we've moved more to SQL and more lower level on the Go side.
Chris: Right. Imagine if you're like, "Oh, I need a list of Pens from the database." There's nothing to help you with that in Go. Go doesn't know what a Pen is. That's just entirely our own data model.
Our goal in life, the end of this API is to have a GraphQL thing that allows you to make a query for Pens. You don't get any of that for free. Nothing. We had to write every line of that.
It was like, "Okay, well, we'll have a resolver for the API, and the API calls some logic because there's some business logic because you might want to sort the Pens in a certain way or something like that. Then the logic will call a database connector."
The database is literally pretty raw SQL.
Dee: Yeah. Yeah.
Chris: And so, that's what you're talking about. It became a little rote to write a SQL statement that says, "Ask the Pens table for Pens ordered in this way - blah-blah-blah."
Dee: Mm-hmm. Mm-hmm. Yeah.
Chris: You do enough rote stuff and then you're like, "Well, this is actually going to be pretty similar for these 37 other tables." [Laughter]
Dee: Go is actually a great language. It's got great reflection features that allow you to look at the language itself and rebuild pieces of rote logic that you're talking about, which is exactly what we did.
We know what all of our DB tables are. We know what all of the columns are. Instead of having to write those models one by one, we have this DB Gen script that creates from the database, ports that over to Go, creates the model that you'll need, every single column that you'll need, and then we found a certain set of queries and updates to the DB that were common for every single model, so a delete, an update, a create, or a find by ID.
All of that is kind of built into Rails. They have common methods like find. You know that it's going to find that record by ID or find by.
And so, what we did was we kind of mimicked those tools that we have on the Rails side, those methods, and just wrote a script that would generate those same query, update, create, delete methods with a set of common inputs. Then we didn't have to write them one by one when we got to that table, when we got to using the collections table or the projects table.
Chris: Yeah. Yeah.
Dee: It's all done in one fell swoop. And any time there's a change to the database, we can rerun that script and have all those pieces built out.
Recently, we just added some new tables for the new product that we're working on. Rachel did it, and she was like, "This DB Gen script is bloody brilliant."
Dee: That is such a compliment from Rachel, like, "It's much more of a joy to work with," and that was something that we brought over from the Rails world. And it helps us move a lot faster.
Chris: Yeah, that's a good point. I didn't make that connection before that it just comes from, you know, we were spoon-fed it before, so let's build a spool to keep spoon-feeding it to ourselves.
The databases are typed. Maybe I made that point already, but I think that's interesting in that, of course, what we want is auto-generated code to help us do all the crud. But because the databases are typed and we know we're in GraphQL land, doing GraphQL is typed too, crucially.
We can get a little bit more than just the queries and such, right? We get, like, "Okay. Well, why don't you auto-generate what the model looks like too? What would a query look like, a typed query?" because you can just look at the columns and be like, "Okay, that's a text column. That's a bullion column. So, I know when I query I should be able to query a bullion on that particular type."
Yeah, this is pretty cool and clever, and help us finish the API quicker because it's still a massive job. It's not like we'll write that tool, run it, and then we could be done in two weeks. Hell no. It's still a very complicated thing.
Importing Rails logic to Go was very often not straightforward.
Chris: Very often it'd be a one-liner in Rails that would translate to 30 lines. Maybe that's an exaggeration sometimes, but sometimes it's not.
Dee: Then we also had to add a whole new permission system, and we had to add authentication that could be shared by Rails and by Go, and then actual authorization piece. We had to build that so that it matched what we already have but then will allow us to build into the future product that we're working on. That was another big reason that things took so long when we thought that it would be super-fast to port the About page.
Dee: But now, going forward, we have a system. We've figured out the permissions piece. It's much easier to use and, hopefully, newer pages will be much, much faster. And as we are building out chunks of the new feature set, it's much easier to do that because we have a proven API, we have proven tools, and it's a joy to work with, so well worth it.
Chris: Yeah, totally. It totally was. It really has.
We haven't rolled it out. As time goes on, we can roll it out to more aspects of the API. That's not really our priority right now because we're just doing other things. But the tooling has proven itself already, like you were saying.
"Oh, we need a new table. Oh, we need APIs, both queries and mutations on a brand new table where a meeting defined the data model yesterday." [Laughter] You know? How long is it going to take to build those APIs? Well, the answer is about half a day or less.
Dee: Yeah. Yeah.
Chris: Because you can just go boop, there it is, and then not only do it but write integration tests for it, too, to make sure you're not going to screw it up. In a way, it's kind of self-documented because, in Go, much like people that are used to working in TypeScript (as I understand it), you just hover over stuff and it just teaches you about the models and stuff that are moving around. It's not like you have to send out a memo to the company explaining how these APIs work. It's just fairly self-documented.
Chris: Very, very satisfying. That's an example of some of the work we are hard at work doing here at CodePen, which it doesn't much matter to the average person just using our beautiful, lovely editor on a day-to-day basis, but it matters for us as we continue to build the future of CodePen.
Dee: Yes. Well said.
Chris: Yeah. Well, thanks for doing so much of this research and so much of the work along the way. There's a little bit more of internal stories about how we were chunking out that work. There was tooling even written to introspect our own APIs so that we could break off chunks of it and assign tickets to each other and stuff.
There's even more to the story here. Because of the fact that we're human beings and we're all working, we needed a way to share the load, as it were. But perhaps another time.
All right, everybody. Take care.
Dee: Have a good one. Thanks, Chris.
Chris: Yeah. Yeah. See ya.
[Radio channel adjustment]