Fast Text To Byte Array Function

I'd like the be able to use the native atob() function, but as I need the data as a byte array rather than text I would then have to convert it. atob() is much faster, taking only 1.2% of the time of the pure JS function. However, converting the decoded text to a byte array is very slow, so that the combination of atob() plus converting to a byte array ends up being almost twice as slow as the pure JS decoder (which converts the data to a byte array as it goes.)This is my current code to convert a string to a byte array. It's probably the simplest code, but it's definitely not fast.?

Code:
var bytearray = Array.prototype.map.call(text, function(x) {return x.charCodeAt(0);});