The MathOverflow mercurial repository.

There is a mercurial repository containing various scripts and data associated with MathOverflow. It’s available at (And since it’s publicly accessible, it’s important to remember not to store secrets in it!)

At this point there’s not too much interesting in there, but some brief highlights:

  • the script that generates trackbacks to the arxiv is in trackbacks/,
  • some documents relating to incorporation are in incorporation/, and
  • our application to the Sloan Foundation for a grant, and some related documents, are in sloan/.

If you need permission to commit changes to the repository, ask the moderators!

How do we generate trackbacks to the arXiv?

MathOverflow generates trackbacks to the arXiv. This means that whenever someone mentions or links to an arXiv paper, we notify the arXiv of this, and they generate a link back to the MathOverflow post. In fact, the arXiv supports trackbacks from a wide variety of sources, including a number of maths blogs. Nevertheless, MathOverflow generates a rather large fraction of the total trackbacks!

You can see a list of recent trackbacks at <>.

The trackbacks are generated by an external script that checks for new or modified posts on MathOverflow, looks through the content for links to the arXiv or arXiv identifiers, and then notifies the arXiv of new links. This all happens  independently of StackExchange, and maintenance of the system is responsibility of the moderators. This purpose of this post is to document this system.

The code that generates trackbacks was originally written by Anton and Scott M, and then partially rewritten by Scott M after the transition to 2.0. It is hosted in the MathOverflow mercurial repository, under the trackbacks/ folder.

The main entry point is the script This should be run at regular intervals (we currently do it hourly), but it is quite robust and can be run more or less frequently, and should cope even with long delays. We currently run this script via a cron job on a machine running on Amazon EC2 (maybe more about that later).

The script itself has two essentially separate components, and two pieces of state.

The state consists of a file timestamp.txt, containing the date of last successful completion of the script, and a file trackbacks containing a list of all previously posted trackbacks. Both of these files are tracked in the mercurial repository, so moving to run on a new machine should be as simple as committing and fetching the repository.

The script is written in an amalgam of bash and python.

The first component of the script, split between and, queries MathOverflow via the API, to find the content of all questions, answers and comments modified since the last successful completion timestamp. (If this script has not run for a while, this can be time consuming, as it requires respecting the throttles requested in the API responses.) It then looks through this content looking for either links to the arXiv or arXiv identifiers. We do this using a series of regular expressions, which allow users to use a variety of different syntaxes! We invested a reasonable amount of time getting this right, going by hand through the database until our regexs seemed to be catching almost everything. If any one ever needs to extract arXiv identifiers from text, is the place to look. Finally we write out the triple

<post-id> <arxiv-id> <post-title>

for each found link, to the temporary file newtrackbacks.

The second component of the script, ./, then reads this output from newtrackbacks, filters out any links which have already been posted according to the state file trackbacks, posts all the new links to the arXiv, and adds them to the trackbacks file.

Finally, the main script runs the two components in turn, and if they both complete successfully updates the  timestamp in timestamp.txt.

It’s possible for this mechanism to miss some links, so we used to also regularly scrape database dumps. Our scripts to do this became obsolete with the transition to 2.0, and no one has fixed them yet.

MathOverflow ( and Gerhard Paseman ) at ICM2014

This is a journal recording promotional events. I am in Seoul to share goings on at ICM2014, the International Congress of Mathematicians. I will be talking up MathOverflow while I am here. Expect the occasional posting on my interests at the congress, which include elementary number theory and general algebra.

Today is Day 0. I will report later about registration, the venue, and other preliminaries to the grand opening tomorrow.

Welcome to the MathOverflow blog!

Hi everyone!


We’ve just started a blog for MathOverflow. We’re not entirely sure where it’s going to go, but we’re getting started with a series of posts by the moderators on the “inner workings” of MathOverflow. These may not be the most exciting things you’ve ever read, but we want to get this information out there for two reasons. Firstly, we think it’s really important that MathOverflow is managed transparently on behalf of the community. We’ve recently established a legal entity for MathOverflow to represent our interests in the contract with Stack Exchange, and in order to be able to handle money donated (thank Sloan Foundation!) for development work. We need to explain this whole setup to the community! Second, as each individual may not stay with the MathOverflow community forever, it’s important that institutional knowledge is recorded and passed on.

Later the blog may evolve into other things — in particular if MathOverflow community members are interested in making that happen, let’s start talking about it!