I've recently had a paper accepted at AJPS and have had to comply with their "replication" policy (which is really a reproduction policy, see this paper for discussion of the difference). This was painful because it required me to give the editors (in a dataverse) my code and data, which they use to reproduce the graphs and other output in my paper without my help outside of the code. This may seem sensible but as research becomes increasingly computational demanding, this policy will become increasingly difficult and unreasonable to comply with.
Interesting research in political science now makes use of annoyingly big data: c.f., some of the work by Fridolin Linder. It took him some months of work on a very expensive and powerful high performance cluster to complete the analyses for this paper. While I was able to modify my code to run on a laptop it simply is not possible in his case. Does that mean that his work can't and shouldn't be published at AJPS?
What exactly is the purpose of this policy? Most optimistically I'd say it is to increase trust. Presumably "verification" by an independent third party means that an article's claims are more trustworthy. I do think this is true but only in a few limited ways. I think this can be illustrated by imagining in what cases an accepted paper would fail this test. It would fail if the code didn't reproduce the outputs contained in the paper. This might occur if I made a mistake in my code that didn't occur in the version that I used to generate the outputs in the paper (or if I was trying to cheat and had fabricated results). However, it wouldn't catch any mistakes that were reflected in the paper, such as any of the most prominent "replication" related scandals in the social sciences (Reinhart and Rogoff, LeCour, Hatemi, etc).
These mistakes could be caught (or made more easy to catch) by requiring transparency by requiring the sharing of code and data (something which AJPS ostensibly does well: I have noticed a number of papers that still don't have their code and data posted). The expending of potentially enormous resources and time in actually re-running everything doesn't really do much though. From what I have been told the Odum Institute at UNC Chapel Hill charges something like $300 per article for "verification." This is a waste!
There are lots of things journals and editors could do to increase trust in analysis that is the result of code.
(1) would help avoid errors made duing "manual" data transformations done in spreadsheet software like those made by Reinhart & Rogoff and Hatemi. (2) would have made it easier to discover LeCour's fraud. (3) and (4) would be a much more comprehensive way to accomplish the ostensible goal of AJPS' current policy. Just because my article is reproducible according to their policy does not mean that the code is free from errors, does what I say it does, or that it will be understandable and useful to others.
Gone are the days where a journal article could contain all of the information necessary to reproduce an analysis and its time that the reviewing process reflected that. It would have been much easier to discover the errors/fraud in the aforementioned cases if the above steps had been taken.
As an addendum, the AJPS policy as it was applied to me actually works to make (1) worse. They required us to write documentation for all source data and remove all columns that weren't used. This decreases reproducibility since now our "source" data files (all of which are public) are now not the same as their public versions. Previously, if one of them were updated you could just replace the old file and our code would clean and transform it as necesary. Not so much now.