build a paragraph-level index for arXiv

There is plenty of technology ready-to-use that does author-topic modeling, in part because so many of the machine-learning community's successes have been in text processing and classification. Some of this technology has been run on the arXiv at the abstract, title, meta-data level, but rarely at the full-text level. If we ran it there, we could build an author-topic model over every paragraph in the full corpus of the arXiv. This would tell us what every paragraph is about (and, amusingly, who wrote every paragraph), with (if we do it right) probabilistic output. That is, it would give probability distributions over what every paragraph is about.

If done correctly, and with a little hand-labeling of classes (and this is easy because each class would have characteristic words and phrases and authors), this could lead to a complete and exceedingly useful paragraph-level index into the arXiv. But even without the hand-labeling it would be incredibly useful: It could be used to find paragraphs in the literature that are useful and relevant to every paragraph you have written in one of your own papers, thus locating related work you might not know about. It could drive search services that find paragraphs relevant to your search terms even when they don't, themselves, contain those terms. And so on!

Dan Foreman-Mackey (NYU) first made it clear to me that this would be possible and he also took some steps towards making it happen. David Blei (Princeton) suggested to me that even from a machine-learning perspective the outcomes could be very interesting. The plagiarism paper from the arXiv people suggests working at PDF level rather than LaTeX source level; I am not sure myself which to do.

emission-line clustering and classification

The BPT diagram has been incredibly productive in classifying galaxies into star-forming and AGN-powered classes. However, the diagram only shows two ratios of nearby lines; ratios of nearby lines so that dust and spectrograph calibration don't mess up the data, and two because it is a single two-dimensional plot. There might be many features in emission-line space sitting undiscovered in the data; there might be many sub-classes and rich structure within the star-forming and AGN groups.

From a data perspective, times have really changed since BPT: (1) There are dozens (well, a dozen) of visible lines in hundreds of thousands of spectra. (2) We have good noise models for the line measurements and this is especially important when they get low in signal-to-noise (as they do if you want to use many lines. (3) We have very well-calibrated spectra now, even spectrophotometrically good to a few percent in the SDSS. (4) The effects of dust attenuation are pretty well understood in the optical. So let's go high dimensional and find all the complex structure that must be there!

The first step is to measure all the lines in a long list, and measure them even when the signal-to-noise is low. We don't care about detections we care about measurements with well-understood noise. The second step is to develop dust-insensitive metrics: What is the distance in data space between two sets of dust-line measurements as a function of noise but marginalizing out the dust affecting each spectrum? Now in that space, let's do some clustering.

I have done nothing on this except discuss it, years ago, with John Moustakas (Siena College). At that time, we were thinking in terms of generating archetypes with an integer program (with my now-deceased guru Sam Roweis). You could use things like support vector machines (great for these kinds of tasks) but we have no labels to classify on. The idea is to find classes not yet discovered! Also SVMs are not sensitive to the uncertainties in the data. I would recommend something like extreme deconvolution which does density estimation of the noise-deconvolved distribution. It can deal with very low signal-to-noise data gracefully. It would have to be modified, however, to project out (marginalize out) the dust-extinction direction in line space. Not impossible but not trivial either.

bimodality search or kurtosis components analysis

Take the SDSS spectra (which are beautifully calibrated spectrophotometrically) and interpolate them onto a common rest-frame (de-redshifted) wavelength grid. Do clever things to interpolate over missing and corrupted data where necessary; this might involve performing a PCA and using the PCA to patch and then re-doing PCA and so on. Then re-normalize the data so that the amplitudes of all the spectra are the same; I am being vague here because I don't know the best choice for definition of amplitude. This is all pre-conditioning for the data; in principle the recommendation here could be applied to any data set; I am just proposing the SDSS spectra.

Now search for a unit-norm (or otherwised normalized) eigenspectrum such that when you dot all pre-conditioned SDSS spectra onto the eigenspectrum, you obtain a distribution of coefficients (dot products) that has minimum kurtosis. That is, instead of finding the principal components—the components with maximum variance—we will look for the platykurtic components—the components with minimum kurtosis. If you are stoked, search the orthogonal subspace for the next-to-minimum kurtosis direction and so on.

Why, you ask? Because low-kurtosis distributions are bi-modal. Indeed, early experiments (performed by Vivi Tsalmantza (MPIA) and myself back in 2008) indicate that this will identify the eigenspectra that best separate the red sequence galaxies from the blue cloud. If you really want to go to town invent a bimodality scalar that is better than kurtosis.

One note: Optimization is a challenge. This sure ain't convex. My approach back in the day was to throw down randomly generated spectra, choose ones that happened to hit fairly low kurtosis, and optimize locally from those.