I'm writing and android app in Kotlin to record and live stream a USB camera video concurrently, both in H264 codec format. I use Google Opensource for WebRTC streaming, and android.media package for recording.
The problem is I have to encode every captured frame twice, since I couldn't find a method to pass H264 encoded frame from recording process to WebRTC. This decrease my app performance largely to a frame rate of only 7-8 fps.
Here is the callback to encode a newly arrived frame in recording thread.
private val mediaEncoderCallback = object : MediaCodec.Callback() {
override fun onInputBufferAvailable(p0: MediaCodec, p1: Int) {}
override fun onError(p0: MediaCodec, p1: MediaCodec.CodecException) {}
private var encoderLock = Object()
override fun onOutputBufferAvailable(
p0: MediaCodec,
index: Int,
info: MediaCodec.BufferInfo
) = synchronized(encoderLock) {
if (isStopped.get()) return
val encodedData = try {
requireNotNull(mediaEncoder?.getOutputBuffer(index))
} catch (exception: Exception) {
Timber.e(exception)
return
}
if (info.flags and MediaCodec.BUFFER_FLAG_CODEC_CONFIG != 0) {
info.size = 0
}
if (info.size != 0) {
if (muxerStarted.get()) {
encodedData.position(info.offset)
encodedData.limit(info.offset + info.size)
val trackId = trackIndex.get()
muxer?.writeSampleData(trackId, encodedData, info)
}
}
mediaEncoder?.releaseOutputBuffer(index, false)
Unit
}
override fun onOutputFormatChanged(p0: MediaCodec, p1: MediaFormat) {
if (isStopped.get()) return
timer.schedule(object : TimerTask() {
override fun run() {
forceStopRecording()
}
}, captureTimeInMillis)
val currentMuxer = muxer ?: return
if (trackIndex.get() == -1) {
currentMuxer.addTrack(mediaEncoder?.outputFormat!!).also {
trackIndex.set(it)
}
}
if (!muxerStarted.get() && trackIndex.get() >= 0) {
muxerStarted.set(true)
currentMuxer.start()
}
}
}
override fun startRecording(captureFile: File, captureTimeInMillis: Long) {
this.captureTimeInMillis = captureTimeInMillis
if (!CodecProvider.checkIfCodecSupported(VIDEO_MIME_TYPE)) return
muxer = MediaMuxer(captureFile.path, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4)
val codec = MediaCodec.createEncoderByType(VIDEO_MIME_TYPE).also {
mediaEncoder = it
}
codec.configure(getVideoFormat(), null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
inputSurface = codec.createInputSurface()
codec.setCallback(mediaEncoderCallback)
codec.start()
codecStarted.set(true)
inputSurface?.let { onSurfaceUpdated(it) }
}
Here is the callback to stream new raw frame in webrtc thread.
override fun handleNewBuffer(cameraKey: String, byteBuffer: ByteBuffer) {
if (!isStreamingStarted) return
val imageArray = ByteArray(byteBuffer.remaining())
byteBuffer.get(imageArray)
byteBuffer.rewind()
val buffer = NV21Buffer(
imageArray,
CameraConfig.WIDTH_SIZE,
CameraConfig.HEIGHT_SIZE,
null
)
val timestamp = TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime())
val videoFrame = VideoFrame(buffer, 0, timestamp)
localVideoSource.capturerObserver.onFrameCaptured(videoFrame)
videoFrame.release()
}
Is there any workaround to encode one time for both streaming and recording (or more specific, to pass *encodedData from recording thread to webrtc thread, instead of a raw byteBuffer).
Thank you!
Related
there is no clear Documentation for playing audio
assets files with the latest Exoplayer
wanted to play multiple audio files simultaneously
and in loop
like Soft Sound
what is the correct way to omit deprecated classes of exoplayer and play audio
build.gradle
// ExoPlayer
api "com.google.android.exoplayer:exoplayer-core:2.18.1"
my
PlayerService.kt file
class PlayerService : Service() {
private val notificationID = 132
private val tag = "Player"
// called when sound is started or stopped
var playerChangeListener: (() -> Unit)? = null
inner class PlayerBinder : Binder() {
fun getService(): PlayerService {
return this#PlayerService
}
}
private val playerBinder = PlayerBinder()
override fun onCreate() {
// load each player into the map
Sound.values().forEach {
exoPlayers[it] = initializeExoPlayer(it.file)
}
}
enum class Sound(val file: String) {
RAIN("rain_sound.ogg"),
...
TABLA("tabla_sound.ogg")
}
private val exoPlayers = mutableMapOf<Sound, ExoPlayer>()
private fun initializeExoPlayer(soundFile: String): ExoPlayer {
// create the player
val trackSelector = DefaultTrackSelector(this)
val exoPlayer = ExoPlayer.Builder(this).setTrackSelector(trackSelector).build()
// load the media source
val dataSource = DefaultDataSourceFactory(this,
Util.getUserAgent(this, this.getString(R.string.app_name)))
val mediaSource = ProgressiveMediaSource.Factory(dataSource)
.createMediaSource(MediaItem.fromUri(Uri.parse("asset:///$soundFile")))
// load the media
Log.d("MAIN", "loading $soundFile")
exoPlayer.setMediaSource(mediaSource)
exoPlayer.prepare()
exoPlayer.play()
// loop indefinitely
exoPlayer.repeatMode = Player.REPEAT_MODE_ALL
return exoPlayer
}
override fun onUnbind(intent: Intent?): Boolean {
// don't continue if we're not playing any sound and the main activity exits
playerChangeListener = null
if (!isPlaying()) {
stopSelf()
Log.d(tag, "stopping service")
}
return super.onUnbind(intent)
}
override fun onBind(intent: Intent?): IBinder {
// return the binding interface
return playerBinder
}
fun startForeground() {
// move to the foreground if we are playing sound
if (isPlaying()) {
....
}
}
fun stopForeground(){...}
fun stopPlaying(){...}
fun isPlaying(): Boolean{...}
fun setVolume(sound: Sound, volume: Float){...}
fun toggleSound(sound: Sound){...}
}
Any help would be highly appreciated.
I have a kotlin multi platform project which contains apollo graphql api
in this project i have BaseRepository Class and in this class there is a method to execute query or mutations
suspend fun <D : Query.Data> executeQuery(query: Query<D>): ApolloResponse<D> {
val response = getApolloClient().query(query).execute()
checkOperation(response)
return response
}
suspend fun <D : Mutation.Data> executeMutation(mutation: Mutation<D>): ApolloResponse<D> {
val response = getApolloClient().mutation(mutation).execute()
checkOperation(response)
return response
}
For example i want to use this method in Some repository like this
class HelpRepository : BaseRepository() {
fun test(request: AddFeedBackRequest) = flow {
val feedBackType = if (request.type == AddFeedBackType.Bug) {
FeedbackType.BUG
} else {
FeedbackType.FEEDBACK
}
val input = AddFeedbackInput(request.note, Optional.presentIfNotNull(feedBackType))
emit(true)
val mutation = AddFeedbackMutation(input)
val response = executeMutation(mutation)
emit(false)
}
}
when i add the flow scope i shouldn't be had to convert this method to a suspend function
i dont want to use suspend function because of ios application. When i use suspend function its convert "Kotlinx_coroutines_coreFlowCollector" in xcode
so i found a wrapper function like this
fun <T> Flow<T>.asCommonFlow(): CommonFlow<T> = CommonFlow(this)
class CommonFlow<T>(private val origin: Flow<T>) : Flow<T> by origin {
fun listen(block: (T) -> Unit): Closeable {
val job = Job()
onEach {
block(it)
}.launchIn(CoroutineScope(Dispatchers.Main + job))
return object : Closeable {
override fun close() {
job.cancel()
}
}
}
}
when i use this wrapper with single variable it works exactly what i want in xcode.
but in functions i couldn't find a proper way to do this
i need a wrapper like
= commonFlow {
}
instead of this
= flow {
}
to use this method as a commonFlow wrapper
Can you help me ?
We have pretty much the same thing in one of our projects. We have a extension function that converts the regular flow to a "common" flow so it can be used in both Android and iOS.
You can created flow like always, and wrap it at the end.
fun <T> Flow<T>.wrap(): CommonFlow<T> = CommonFlow(this)
class HelpRepository : BaseRepository() {
fun test(request: AddFeedBackRequest) = flow {
val feedBackType = if (request.type == AddFeedBackType.Bug) {
FeedbackType.BUG
} else {
FeedbackType.FEEDBACK
}
val input = AddFeedbackInput(request.note, Optional.presentIfNotNull(feedBackType))
emit(true)
val mutation = AddFeedbackMutation(input)
val response = executeMutation(mutation)
emit(false)
}
}.wrap()
This question already has answers here:
Android - implementing startForeground for a service?
(11 answers)
Closed 11 months ago.
I've createde a simple app (following this link How to Play Sound On Button Click in Android Studio Java 2021 ), and everything works fine, but, i need this app can still playing the sound after minimize then or block the phone (is a white noise app from help on sleep for my baby daughter, and i need that keep playing all night).
Is there any way or setting that allows the sound not to stop after minimizing or blocking the application?
Thank you for enveryone that could help me and my baby :)
If you add the ability to run your application in the background, it will solve your problem. Resources you can review:
https://developer.android.com/guide/components/services
https://developer.android.com/guide/background/threading
Basic Example :
AndroidManifest.xml :
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<application
*
*
*
*
<service android:name=".ui.main.AudioPlayerService"/>
</application>
AudioPlayerService :
private const val PLAYBACK_CHANNEL_ID = "blabla"
private const val PLAYBACK_NOTIFICATION_ID = 1
class AudioPlayerService : Service() {
private var player: SimpleExoPlayer? = null
private var playerNotificationManager: PlayerNotificationManager? = null
private val mediaItem: MediaItem = MediaItem.fromUri(ApiInterface.Channel24LiveURL)
/** Classes to be connected to the service with the
service
*reference providing the link between
*/
private val mBinder = AudioServiceBinder()
/**
* Data that the service will share to other classes. */
inner class AudioServiceBinder : Binder() {
val service
get() = this#AudioPlayerService
val player
get() = this#AudioPlayerService.player
}
override fun onBind(intent: Intent?): IBinder? {
return mBinder
}
/**
* Service start part
*/
override fun onCreate() {
super.onCreate()
// init player
player = SimpleExoPlayer.Builder(this)
.build().apply {
setMediaItem(mediaItem)
playWhenReady = true
prepare()
}
playerNotificationManager = PlayerNotificationManager.createWithNotificationChannel(
applicationContext,
PLAYBACK_CHANNEL_ID,
R.string.playback_channel_name,
R.string.playback_channel_desc,
PLAYBACK_NOTIFICATION_ID,
object : PlayerNotificationManager.MediaDescriptionAdapter {
override fun getCurrentContentTitle(player: Player): CharSequence {
return getString(R.string.def_playback_title)
}
override fun createCurrentContentIntent(player: Player): PendingIntent? {
return PendingIntent.getActivity(
applicationContext,
0,
Intent(applicationContext, MainActivity::class.java),
PendingIntent.FLAG_CANCEL_CURRENT
)
}
override fun getCurrentContentText(player: Player): CharSequence? {
return null
}
override fun getCurrentLargeIcon(
player: Player,
callback: PlayerNotificationManager.BitmapCallback
): Bitmap? {
return getBitmapFromVectorDrawable(applicationContext, R.mipmap.ic_launcher)
}
}, object : PlayerNotificationManager.NotificationListener {
override fun onNotificationCancelled(notificationId: Int, dismissedByUser: Boolean) {
stopSelf()
}
override fun onNotificationPosted(notificationId: Int, notification: Notification, ongoing: Boolean) {
if (ongoing) {
// Make sure the service will not get destroyed while playing media.
startForeground(notificationId, notification)
} else {
// Make notification cancellable.
stopForeground(false)
}
}
}
).apply {
// previous and next actions.
setUseNavigationActions(true)
setPlayer(player)
}
}
#MainThread
private fun getBitmapFromVectorDrawable(
context: Context,
#Suppress("SameParameterValue") #DrawableRes drawableId: Int
): Bitmap? {
return ContextCompat.getDrawable(context, drawableId)?.let {
val drawable = DrawableCompat.wrap(it).mutate()
val bitmap = Bitmap.createBitmap(
drawable.intrinsicWidth,
drawable.intrinsicHeight,
Bitmap.Config.ARGB_8888
)
val canvas = Canvas(bitmap)
drawable.setBounds(0, 0, canvas.width, canvas.height)
drawable.draw(canvas)
bitmap
}
}
#MainThread
fun changePlayerVolume(view: ImageView) {
player?.let {
if (it.volume == 1.0f) {
it.volume = 0.0f
view.setImageDrawable(ContextCompat.getDrawable(this, R.drawable.ic_volume_off))
} else {
it.volume = 1.0f
view.setImageDrawable(ContextCompat.getDrawable(this, R.drawable.ic_volume_up))
}
}
}
#MainThread
fun jumpLiveStream() {
player?.let {
it.setMediaItem(mediaItem)
it.playWhenReady = true
}
}
/**
* Triggered when the app is closed. */
override fun onTaskRemoved(rootIntent: Intent?) {// Stop the service when the user closes the application.l
releasePlayer()
stopSelf()
super.onTaskRemoved(rootIntent)
}
override fun onDestroy() {
releasePlayer()
stopSelf()
super.onDestroy()
}
/**
* delete player and playerNotificationManager */
private fun releasePlayer() {
player?.let {
it.release()
player = null
}
playerNotificationManager?.let {
it.setPlayer(null)
playerNotificationManager = null
}
}
}
Good luck!
I need to develop an app which monitors a list of apps and, if the system is rebooted, return to the last open page of the monitored apps. So, say that android was on the youtube app playing lofi hip hop radio - beats to relax/study to. How could my app send android back to this given video? Also, is there a way for me to know where in the app the user is? For example, can I know that the user was on youtube playing lofi and open it back again for them?
I already know how to open another app as well as detect whichever app is currently open, but I need to know the its uri as well. I can use adb and shell script if it is needed.
I have the following code to launch another app:
class MainActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
intent.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
val intent = Intent(this, BackAppListenerService::class.java)
startService(intent)
openApp(this, "com.google.android.youtube")
}
/** Open another app.
* #param context current Context, like Activity, App, or Service
* #param packageName the full package name of the app to open
* #return true if likely successful, false if unsuccessful
*/
fun openApp(context: Context, packageName: String?): Boolean {
val manager = context.packageManager
return try {
val i = manager.getLaunchIntentForPackage(packageName!!)
if (i == null) {
println("Activity not found")
return false;
//throw new ActivityNotFoundException();
}
//throw new ActivityNotFoundException();
i.addCategory(Intent.CATEGORY_LAUNCHER)
context.startActivity(i)
true
} catch (e: ActivityNotFoundException) {
println(e)
false
}
}
}
And with this I get the current foreground app
class BackAppListenerService : Service() {
private var isRunning = false
private var lastApp = ""
#RequiresApi(Build.VERSION_CODES.LOLLIPOP)
override fun onCreate() {
isRunning = true
Intent(Settings.ACTION_USAGE_ACCESS_SETTINGS)
}
override fun onStartCommand(intent: Intent, flags: Int, startId: Int): Int {
//Creating new thread for my service
//Always write your long running tasks in a separate thread, to avoid ANR
Thread(Runnable {
while (true) {
try {
Thread.sleep(10)
} catch (e: Exception) {
}
val currentForegroundApp = getForegroundApp()
val currentApp = currentForegroundApp.first
if (currentApp != lastApp) {
// New app on front
lastApp = currentApp
println("Current App $lastApp")
}
}
}).start()
return START_STICKY
}
// Must Have Usage Access Permission
fun getForegroundApp(): Pair<String, UsageStats> {
var currentApp = "NULL"
var currentAppInfo: UsageStats? = null
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
val usm = this.getSystemService(Context.USAGE_STATS_SERVICE) as UsageStatsManager
val time = System.currentTimeMillis()
val appList =
usm.queryUsageStats(UsageStatsManager.INTERVAL_DAILY, time - 1000 * 1000, time)
if (appList != null && appList.size > 0) {
val mySortedMap: SortedMap<Long, UsageStats> =
TreeMap<Long, UsageStats>()
for (usageStats in appList) {
mySortedMap.put(usageStats.lastTimeUsed, usageStats)
}
if (mySortedMap != null && !mySortedMap.isEmpty()) {
currentAppInfo = mySortedMap[mySortedMap.lastKey()]!!
currentApp = mySortedMap[mySortedMap.lastKey()]!!.packageName
}
}
} else {
val am = this.getSystemService(Context.ACTIVITY_SERVICE) as ActivityManager
val tasks = am.runningAppProcesses
currentApp = tasks[0].processName
}
return Pair(currentApp.split(".").last(), currentAppInfo!!)
}
[...]
}
Any help is much appreciated.
If I understand well, once ARCore 1.0 will be released on Google Play, it will be necessary to install it on the device in order to be able to run an ARCore app.
How to check if ARCore lib/apk is installed on device ?
Should be sufficient to do something like:
try {
arCoreSession = Session(this)
val config = Config(arCoreSession)
if (!arCoreSession.isSupported(config)) {
Logger.d("ARCore not installed")
} else {
arCoreSession.configure(config)
}
} catch (ex: Throwable) {
Logger.d("ARCore not installed")
}
This is what I'm using here for one my apps and works fine on devices with or wothout ARCore.
According to ARCore documentation 1.4.0, if optional it is important check its availability recursively and then install it:
void maybeEnableArButton() {
// Likely called from Activity.onCreate() of an activity with AR buttons.
ArCoreApk.Availability availability = ArCoreApk.getInstance().checkAvailability(this);
if (availability.isTransient()) {
// re-query at 5Hz while we check compatibility.
new Handler().postDelayed(new Runnable() {
#Override
public void run() {
maybeEnableArButton();
}
}, 200);
}
if (availability.isSupported()) {
mArButton.setVisibility(View.VISIBLE);
mArButton.setEnabled(true);
// indicator on the button.
} else { // unsupported or unknown
mArButton.setVisibility(View.INVISIBLE);
mArButton.setEnabled(false);
}
}
If already supported just check if ARCore is installed:
// Set to true ensures requestInstall() triggers installation if necessary.
private boolean mUserRequestedInstall = true;
// in onResume:
try {
if (mSession == null) {
switch (ArCoreApk.getInstance().requestInstall(this, mUserRequestedInstall)) {
case INSTALLED:
mSession = new Session(this);
// Success.
break;
case INSTALL_REQUESTED:
// Ensures next invocation of requestInstall() will either return
// INSTALLED or throw an exception.
mUserRequestedInstall = false;
return;
}
}
} catch (UnavailableUserDeclinedInstallationException e) {
// Display an appropriate message to the user and return gracefully.
return;
} catch (...) { // current catch statements
...
return; // mSession is still null
}
Sometimes it is easier to request this with Rx methodology. Here's the code:
private fun getArAvailabilityRx(context: Context): Single<ArCoreApk.Availability> {
return Single.fromCallable<ArCoreApk.Availability> {
ArCoreApk.getInstance().checkAvailability(context)
}.flatMap { availability ->
if (availability.isTransient) {
// `isTransient` means it hasn't finished loading value; let's request the value in 500 ms
getArAvailabilityRx(context).delaySubscription(500, TimeUnit.MILLISECONDS, AndroidSchedulers.mainThread())
} else {
Single.just(availability)
}
}.observeOn(AndroidSchedulers.mainThread())
}
Here's a little utility class I wrote (based originally on something from https://github.com/google/helloargdx).
It will perform all the checks and setup necessary, in order to ensure it is safe to launch a Session.
abstract class ArCheckFragment : Fragment() {
private var userRequestedInstall = true
abstract fun onCameraPermissionDeny()
abstract fun onArCoreUnavailable(availability: Availability)
abstract fun onArCoreInstallFail(exception: UnavailableException)
abstract fun onArCoreInstallSuccess()
override fun onResume() {
super.onResume()
performCheck()
}
override fun onRequestPermissionsResult(
requestCode: Int,
permissions: Array<String>,
grantResults: IntArray
) {
super.onRequestPermissionsResult(requestCode, permissions, grantResults)
if (requestCode == REQUEST_CODE_CAMERA_PERMISSION) {
for (i in permissions.indices) {
if (permissions[i] == Manifest.permission.CAMERA &&
grantResults[i] == PackageManager.PERMISSION_GRANTED
) {
checkArCore()
return
}
}
onCameraPermissionDeny()
}
}
/**
* Performs the whole check
*/
fun performCheck() {
if (requestCameraPermission()) {
checkArCore()
}
}
/**
* Requests the camera permission, if necessary.
* #return whether camera permission is already granted. If so, the permission won't be requested.
*/
private fun requestCameraPermission(): Boolean {
if (ContextCompat.checkSelfPermission(
requireContext(),
Manifest.permission.CAMERA
) == PackageManager.PERMISSION_GRANTED
) {
return true
}
requestPermissions(arrayOf(Manifest.permission.CAMERA), REQUEST_CODE_CAMERA_PERMISSION)
return false
}
private fun checkArCore() {
if (!isResumed) {
return
}
val availability = ArCoreApk.getInstance().checkAvailability(activity)
if (availability.isTransient) {
requireView().postDelayed(AR_CORE_CHECK_INTERVAL) { checkArCore() }
return
}
when (availability) {
Availability.SUPPORTED_INSTALLED ->
onArCoreInstallSuccess()
Availability.SUPPORTED_APK_TOO_OLD,
Availability.SUPPORTED_NOT_INSTALLED ->
startArCoreInstallation()
else ->
onArCoreUnavailable(availability)
}
}
private fun startArCoreInstallation() {
try {
val installStatus =
ArCoreApk.getInstance().requestInstall(activity, userRequestedInstall)
when (installStatus) {
InstallStatus.INSTALLED -> onArCoreInstallSuccess()
InstallStatus.INSTALL_REQUESTED,
null ->
// Ensures next invocation of requestInstall() will either return
// INSTALLED or throw an exception.
userRequestedInstall = false
}
} catch (exception: UnavailableException) {
onArCoreInstallFail(exception)
}
}
companion object {
private const val REQUEST_CODE_CAMERA_PERMISSION = 1
private const val AR_CORE_CHECK_INTERVAL = 200L
}
}
You can subclass this Fragment and implement the abstract functions to receive callbacks on what the result of these checks is. Only in onArCoreInstallSuccess is it safe to create a Session.