I too think James died before the 1850 Census. Virginia wasn't recording deaths that early. I'd contact the Chesterfield Historical Society of Virginia and ask if they have a local death index, newspaper death notices and cemetery inventories.
I'd also look at the county records on microfilm at the Library of Virginia. You should find his probate records and guardian court records for the children.
Also look for the original of James and Amanda's marriage record. Both sets of parents might be on it. If not, look for the newspaper announcement of their marriage. Both sets of parents might be in it.
Amanda S. Rudd died 13 Dec 1891 in Chesterfield County. I'd look for her newspaper death notice to see if her children are mentioned. Successful Searching!