Search functionality on CodePen just got a huge upgrade. You can see for yourself by checking out this search for rainbows!

We decided to bring this project in-house, so now we do all our own search indexing, algorithms, and all that fancy stuff. The result is a very CodePen-like experience. Running a search will return a grid of Pens like you are used to from many other views on CodePen:

Search-Screen

It works just like you would expect. You type in search terms, submit the form, and you will be displayed Pens that are relevant to your search. We try to give you both quality and quantity. By default, the results are ordered by our own special algorithm, which is similar to how we measure popularity. The most relevant and popular Pens are displayed first, but we also match everything we can so you can paginate as far back as you wish.

Advanced Search Features

Clicking the "Advanced Options" link on the search page reveals dropdowns to narrow down your search or change the focus.

advanced-options

Brand new is the ability to search only your Pens or only the Pens of people you follow. I know personally I have hundreds and hundreds of Pens, and searching to find them is much quicker than paginating through them, even with list view. Often when you're searching you have a specific Pen in mind. If you know you follow the person, narrowing the search only to them can make the search much easier.

Also, if you know the title of the Pen you are looking for, narrowing the search to titles only will make that much easier to find. Changing the ordering to list newest-first can be similarly useful in tracking down a Pen a specific Pen when you know about when it was created.

User Search

As we covered, you can search your own Pens now. That ability it also exposed right on your profile page.

search-users

You can search your own Pens, which will include your Private Pens (if you are logged in). Other people can search your Pens as well, which of course will not include Private Pens.

A Far Cry From The Old Days

Search functionality was previously simply a Google Custom Search form. While Google is without question the master of search, it actually didn't do a spectacular job searching Pens. For one thing, it didn't search the actual code very well. Our new search does. For another, you couldn't see a preview of the Pen itself from the results page. You just had to guess from the title and description, which is no good.

Not to mention the fact that it is cost prohibitive to use without Google displaying ads on it, which surely nobody likes. Also not to mention that Google couldn't do the advanced search stuff like we described above.

Long live new search!

Behind The Scenes

What looks like a simple feature is actually quite complex. This feature involves special servers, tokenizing and indexing of data, queuing, large fancy open source projects, and a host of other things that make my front-end boots shake.

Our server master Tim Sabat explains:

We use Lucene / Solr running on Tomcat 6.0 as our search server on AWS. We use Resque to queue up indexing requests. Then standard HTTP requests to query the docs.

We can index about 15 documents per second.

nerd-soup

The queuing part is interesting and important. We need to index Pens when they are created and re-index them when they are changed and saved. We could just directly ask the search server to do those things when those actions happen, but that's not very smart. If that server is down, slow, or backed up, those requests could get lost and thus our search index incomplete. Instead we save all those requests to a queue and the queue processes them as it is able so no request ever gets lost.

Our Top 10 Search Terms

Just for fun:

  1. menu
  2. button
  3. slider
  4. form
  5. loading
  6. login
  7. parallax
  8. hover
  9. navigation
  10. loader

Enjoy!

Of course the best part about good search is that you can find what you are looking for fast. That's what we want to enable with new search. We hope CodePen is a great place to come to look for examples of design and front end engineering when you need some help or inspiration.