Updated by Don Koks, 2012.
Original by Philip Gibbs and Jim Carr, late 1990s.
The concept of mass has always been fundamental to physics. It was present in the earliest days of the subject, and its importance has only grown as physics has diversified over the centuries. Its definition goes back to Galileo and Newton, for whom mass was that property of a body that enables it to resist externally imposed changes to its motion. Newton used mass to define momentum and force vectors: he defined a body's momentum as p = mv (where v is its velocity), and he defined force to be the rate of increase of the body's momentum: F = dp/dt. When a body's mass is constant (as it usually is, except when we are analysing the motion of e.g. a rocket), the force law becomes F = m dv/dt = ma, where a is the body's acceleration.
This definition of mass was applied in a straightforward way for almost two centuries. Then Einstein arrived on the scene and, in his theory of motion known as special relativity, the situation became more complicated. The above definition of mass still holds for a body at rest, and so has come to be called the body's rest mass, denoted m0 if we wish to stress that we're dealing with rest mass. But when the body is moving we find that its force–acceleration relationship now depends on two quantities: the body's speed, and the angle between its direction of motion and the applied force. When we relate the force to the resulting acceleration along each of three mutually perpendicular spatial axes, we find that in each of the three expressions a factor of γm0 appears, where the gamma factor γ = (1–v2/c2)–1/2 occurs frequently in special relativity.
The idea of a speed-dependent mass actually dates back to Lorentz's work. His 1904 paper Electromagnetic Phenomena in a System Moving With Any Velocity Less Than That of Light introduced the "longitudinal" and "transverse" electromagnetic masses of the electron. With these he could write the equations of motion for an electron in an electromagnetic field in the newtonian form, provided the electron's mass was allowed to increase with its speed. Between 1905 and 1909, the relativistic theory of force, momentum, and energy was developed by Planck, Lewis, and Tolman. It turned out that a single mass dependence could be used for any acceleration, thus enabling mass to retain its independence of the body's direction of acceleration, if a speed-dependent "relativistic mass" m was understood as present in Newton's original expression p = mv.
So a body moving with speed v and whose momentum has magnitude p has a relativistic mass given by m = p/v, and (it turns out) a total energy of mc2. A body with rest mass m0 turns out to have relativistic mass γm0. But the definition m = p/v now also neatly defines a relativistic mass for a photon: this moves with speed c and has energy E, and electromagnetic theory gives it a momentum of magnitude p = E/c, so it has relativistic mass p/v = E/c2. The expression m = γm0 doesn't apply to a photon, for which γ is infinite.  But on the other hand, writing m = γm0 won't lead to any contradictions for a photon if we define the photon's rest mass to be zero. See the FAQ article What is the mass of a photon? for more discussion of this.
It seems to have been Lewis who introduced the appropriate speed dependence of mass in 1908, but the term "relativistic mass" appeared later. (Gilbert Lewis was a chemist whose other claim to fame in physics was naming the photon in 1926.) Relativistic mass came into common usage in the relativity texts of the early 1920s written by Pauli, Eddington, and Born. But whereas rest mass is routinely used in many areas of physics, relativistic mass is mostly restricted to the dynamics of special relativity. Because of this, a body's rest mass tends to be called simply its "mass".
The quantities that a moving observer measures as scaled by γ in special relativity are not confined to mass. Two others commonly encountered in the subject are a body's length in the direction of motion and its ageing rate, both of which get reduced by a factor of γ when measured by a passing observer. So, a ruler has a rest length, being the length it was given on the production line, and a relativistic or contracted length in the direction of its motion, which is the length we measure it to have as it moves past us. Likewise, a stationary clock ages normally, but when it moves it ages slowly by the gamma factor so that its "factory tick rate" is reduced by γ. Lastly, an object has a rest mass, being the mass it "came off the production line with", and a relativistic mass, being defined as above. When at rest, the object's rest mass equals its relativistic mass. When it moves, its acceleration is determined by both its relativistic mass (or its rest mass, of course) and its velocity.
The use of these γ-scaled quantities is governed only by the extent to which they are useful. While contracted length and time intervals are used—or not—insofar as they simplify special relativity analyses, relativistic mass has found itself at the centre of much debate in recent years about whether it is necessary in a physics curriculum. All physicists use rest mass, but not all physicists would have relativistic mass appear in textbooks, preferring instead always to write it in terms of rest mass when it is used (although this can't be done for photons). So, if all physicists agree that rest mass is a very fundamental concept, then why use relativistic mass at all?
When particles are moving, relativistic mass provides a very economical description that absorbs the particles' motion naturally. For example, suppose we put an object on a set of scales that are capable of measuring incredibly small increases in weight. Now heat the object. As its temperature rises causing its constituents' thermal motion to increase, the reading on the scales will increase. If we prefer to maintain the usual idea that mass is proportional to weight—assuming we don't step onto an elevator or change our home planet midway through the experiment—then it follows that the object's mass has increased. If we define mass in such a way that the object's mass does not increase as it heats up, then we will have to give up the idea that mass is proportional to weight.
Another many-particle example occurs in pre-relativistic physics, in which the centre of mass of an object is calculated by "weighting" the position vector ri of each of its particles by their mass mi:
∑i miri Centre of mass = ———————— ∑i miThe same expression will hold relativistically if each of the above masses is now a particle's relativistic mass. If we prefer to use only rest mass then we must replace the mi in the above expression by γi mi where mi is rest mass, but now the expression has lost a certain economy. Similarly, if two objects with relativistic masses m1 and m2 collide and stick together in such a way that the resulting object is at rest, then its (relativistic = rest) mass will be m1 + m2. This accords with our intuition, and intuition is mostly what good conventions are about. In contrast, a rest-mass-only analysis describes the interaction by saying that the objects have (rest) masses of M1 and M2, with a combined (rest) mass of γ1M1 + γ2M2. Whether our intuition has anything to gain from this new expression is not clear.
Another place where the idea of relativistic mass surfaces is when describing the cyclotron, a device that accelerates charged particles in circles within a constant magnetic field. The cyclotron works by applying a varying electric field to the particles, and the frequency of this variation must be tuned to the natural orbital frequency that the particles acquire as they move in the magnetic field. But in practice we find that as the particles accelerate, they begin to get out of step with the applied electric field and can no longer be accelerated further. This can be described as a consequence of their masses increasing, which changes their orbital frequency in the magnetic field.
Lastly, the energy E of an object, whether moving or at rest, is given by Einstein's famous relation E = mc2, where m is its relativistic mass. Because, for example, the photon has no rest mass but does have relativistic mass, the use of relativistic mass makes it much easier to describe the mass changes that happen when light interacts with matter. See the FAQ article What is the mass of a photon?.
While relativistic mass is useful in the context of special relativity, it is rest mass that appears most often in the modern language of relativity, which centres on "invariant quantities" to build a geometrical description of relativity. Geometrical objects are useful for unifying scenarios that can be described in different coordinate systems. Because there are multiple ways of describing scenarios in relativity depending on which frame we are in, it is useful to focus on whatever invariances we can find. This is, for example, one reason why vectors (i.e. arrows) are so useful in maths and physics; everyone can use the same arrow to express e.g. a velocity, even though they might each quantify the arrow using different components because each observer is using different coordinates. So the reason rest mass, rest length, and proper time find their way into the tensor language of relativity is that all observers agree on their values. (These invariants then join with other quantities in relativity: thus, for example, the four-force acting on a body equals its rest mass times its four-acceleration.) Some physicists cite this view to maintain that rest mass is the only way in which mass should be understood.
As with many things, the use of relativistic mass can be a matter of taste, but it seems that at least some physicists who vehemently oppose the use of relativistic mass believe, mistakenly, that pro-relativistic mass physicists are against the idea of rest mass. It's not clear just why there should be this perennial confusion about preferences, and why some of those who dislike the idea of relativistic mass show such fundamentalist opposition to a choice of formalism that can never produce wrong results. The world of physics and its language is full of useful alternative notations and ways of approaching things, and different choices of notation and language can shed light on the physics involved. Selecting one of the other of relativistic versus rest mass will never lead to problems for practitioners of the subject.
A debate of the subject surfaced in Physics Today in 1989 when Lev Okun wrote an article urging that relativistic mass should no longer be taught [1]. Wolfgang Rindler responded with a letter to the editor defending its continued use [2]. In 1991 Tom Sandin wrote an article in the American Journal of Physics that argued in favour of relativistic mass [3].
A commonly heard argument against the use of relativistic mass runs as follows: "The equation E = mc2 says that a body's relativistic mass is proportional to its total energy, so why should we use two terms for what is essentially the same quantity? We should just stay with energy, and use the word 'mass' to refer only to rest mass." The first difficulty with this line of reasoning is that it is quite selective; after all, it should surely rule out the use of rest mass as well, since within special relativity, rest mass is proportional to a body's rest energy. On that note, a second difficulty of the line of reasoning is more technical: equating energy and relativistic mass cannot be done more generally. In general relativity, it's natural to consider quantities that are conserved for a system moving on a geodesic. But γ m is not generally conserved along geodesics. (Actually, γ m is called pt in the language of general relativity. It turns out that a closely related quantity, pt, will be conserved along a geodesic if the metric is time independent.) Note, though, that while relativistic mass γ m is not a body's total energy in general relativity, it's also not simply the source of gravity within the same theory. Finally, a third difficulty with the above commonly heard argument is that, in the interests of consistency, it should surely be applied to rule out either the "momentum density" or the "energy flux density" of light, since these also are simply related by a factor of c2. Yet, and quite rightly, these last two terms co-exist in modern literature; no one ever suggests that either of these terms should be dropped in favour of the other, because they both have their uses and are fundamentally different quantities: a volume density and an areal density.  The fact that they can be equated is the interesting thing physically, but one must not thereafter trivialise the physics by insisting that one of the two concepts be dropped.
So likewise can the concepts of mass and energy coexist. The above argument that E = mc2 demotes mass in favour of energy—or rather, that it selectively demotes relativistic mass, but not rest mass—also neglects the very definitions of mass and energy. Mass is a property of a body that we have an intuitive feel for; its definition as a resistance to acceleration is very fundamental. Energy, on the other hand, is defined in physics in a technical way that involves the concept of a system's time evolution; this is not something that bears any obvious similarity to the concept of an object's resistance to being accelerated. If the concept of mass exists in some sense "prior" to that of energy, and if energy itself is defined in a different way to mass, then it does not seem reasonable to drop the idea of mass in favour of energy. Rather, E = mc2 becomes an expression that tells us how much energy a given mass has; it also tells us how much a body will resist being accelerated depending on its energy content. And, perhaps best of all, it reminds us that Einstein's equation is a triumph of relating two disparate quantities—and this is one of the great aims of physics.
Another argument sometimes put forward for dropping the use of relativistic mass is that since e.g. all electrons have the same rest mass (whereas their relativistic masses depend on their speeds), then their rest mass is the only quantity able to be tabulated, and so we should discard the very idea of relativistic mass. But when we say without qualification that "the height of the Eiffel Tower is 324 metres", we clearly mean its rest length; but that doesn't mean the idea of contracted length should be discarded. Similarly, it's okay to say that the mass of an electron is about 10–30 kg without having to specify that we are referring to the rest mass; everyone knows we mean rest mass when we tabulate a particle's mass. That's purely a useful linguistic convention, and it does not imply that we have discarded the idea of relativistic mass, or that it should be discarded at all.
Everyone agrees that a moving train's rest mass is a fixed property inherent to it, just as its rest length is a fixed property inherent to it. And yet, strangely, many of the same physicists who insist that a moving train's mass does not scale by γ are quite happy to say that its length does scale by γ. There is no argument in the literature about the uses of rest length versus moving length, so why should there be any argument about the uses of rest mass versus moving mass?
Another mass concept that everyone agrees on is the idea of reduced mass in non-relativistic mechanics. When the mechanics of e.g. a sun–satellite system or a mass oscillating on a spring is analysed, a mass term appears that combines the two masses in a particular way. As far as the maths goes, it's as if we are replacing the two original bodies by two new ones: the first new body has infinite mass, and the second new body has a mass equal to the system's reduced mass, which has this name because it's smaller than either of the two original masses that gave rise to it. This is a fruitful way to view the original system, and it's completely standard. No one gets confused into thinking that we actually have an infinite mass and a reduced mass in our system. No one worries that the new, infinite, mass is somehow going to become a black hole, or that the reduced mass lost some of its atoms somewhere. Everyone knows the realm of applicability of the concept of reduced mass and how useful it is. Why then, do so many physicists criticise relativistic mass by squeezing it into realms where it was never intended to be used? They presumably don't do the same thing with reduced mass.
An optimistic view would hold that it's a measure of the richness of physics that focussing on different aspects of concepts like mass produces different insights: intuition in the case of relativistic mass in special relativity, and the also-intuitive notion of invariance and geometrical quantities in the case of rest mass within the tensor language of special and general relativity. The two aspects do not contradict each other, and there is room enough in the world of physics to accommodate them both.
Abandoning the use of relativistic mass is sometimes validated by quoting select physicists who are or were against the term, or by exhaustively tabulating which textbooks use the term. But real science isn't done this way. In the final analysis, the history of relativity, with its quotations from those in favour of relativistic mass and those against, has no real bearing on whether the idea itself has value. The question to ask is not whether relativistic mass is fashionable or not, or who likes the idea and who doesn't; rather, as in any area of physics notation and language, we should always ask "Is it useful?". And relativistic mass is certainly a useful concept. There can be little doubt that some of its vocal opponents even use it to gain intuition when analysing a scenario in special relativity.
The concept of relativistic mass is neatly encapsulated in the expression F = d(mv)/dt, where m is relativistic mass. This says that an impulse F dt causes an infinitesimal increase in a body's relativistic momentum mv.
Besides this definition and use of relativistic mass, we wish here to write down the relativistic version of Newton's second law, F = ma. In Newton's mechanics, this equation relates vectors F and a via the mass m of the object being accelerated, which is invariant in Newton's theory. Because m is just a number, in Newton's theory the force on an object is always parallel to the resulting acceleration.
The corresponding equation in special relativity is a little more complicated. It turns out that the force F is not always parallel to the acceleration a. We can express this fact using matrix notation. Let m be the rest mass, and v be the velocity as a column vector, whose entries are expressed as fractions of c and whose magnitude v is the speed as a fraction of c. Let vt be the velocity as a row vector, and let 1 be the 3 × 3 identity matrix. As usual, set γ = (1 – v2) –1/2. The relativistic version of F = ma turns out to be
F = (1 + γ2 v vt) γ m a and a = (1 – v vt) F —————————— γ mSo defining mass via force and acceleration isn't as simple as it was for Newton (although it is simple, in principle, to define the mass as relating impulse and momentum increase, as mentioned a few lines up). Nevertheless, the three components of the two expressions above share a factor of γ m, and the rest mass m only ever appears in both expressions accompanied by γ. The acceleration is not necessarily parallel to the force that produced it, and it's not hard to see from the above equations that it's easier to accelerate a mass sideways to its motion than it is to accelerate it in the direction of its motion. This is how relativity reproduces Lorentz's original concepts of longitudinal and transverse masses; they are actually contained in these equations. The directional dependence that the newtonian meaning of mass has now taken on is neatly contained in the matrices 1 + γ2 v vt and 1 – v vt, and the remaining factor γ m is the relativistic mass. Taking our cue from the equations like this, to isolate quantities that might prove useful, is a powerful tool in mathematical physics.
[1] The Concept of Mass, Physics Today, 42 June 1989, pg 31
[2] Putting to Rest Mass Misconceptions, Physics Today 43, May 1990, pgs 13 and 115
[3] In Defense of Relativistic Mass, Am. J. Phys. 59, November 1991, pg 1032
Some historical details can be found in Concepts of Mass by Max Jammer and Einstein's Revolution by Elie Zahar.