I am trying to make an AI to play simple Snake game following this article. With my current solution snake is able to come up to score of around 35 points. For some reason it will handle the walls correctly but would almost every time die by hitting its own tail.
I am not able to pinpoint my mistake, but I assume its the way I use fit function on the neural network.
for (int i = 0; i < EPOCHS; i++) {
epsilon = 0.9;
board.initGame();
State state = board.getState();
while (state.isInGame()) {
// Get next action to perform
final Action action = epsilonGreedyAction(state);
// Play action and get reward
final double reward = board.getRewardForAction(state, action);
// Get next state
final State nextState = board.getState();
// Update neural network
update(state, action, reward, nextState);
// Apply next state
state = nextState;
}
}
private Action epsilonGreedyAction(final State state) {
epsilon -=0.001;
double random = SnakeUtil.getRandom();
if (random < epsilon) {
randomActions++;
return Action.randomAction(state);
}
return SnakeUtil.getMaxQAction(state, Q_TABLE);
}
getMaxQAction will return any action that is in the Q_TABLE for this state that has had high reward up until now (max reward).
Q_TABLE is updated also in the update method
private void update(final State state, final Action action, final double reward, final State nextState) {
MaxQ maxQ = SnakeUtil.getMaxQ(nextState, Q_TABLE);
double targetReward = reward + (0.90 * maxQ.getReward());
SnakeUtil.updateQTable(state, action, targetReward, Q_TABLE);
net.fit(buildObservation(state).getData(), Nd4j.create(fromAction(action)));
}
private double[][] fromAction(final Action action) {
return new double[][] {
{
action.equals(Action.UP) ? 1 : 0,
action.equals(Action.RIGHT) ? 1 : 0,
action.equals(Action.DOWN) ? 1 : 0,
action.equals(Action.LEFT) ? 1 : 0
}
};
}
private Observation buildObservation(final State state) {
return new Observation(Nd4j.create(new boolean[][]{
{
state.isCanGoUp(),
state.isCanGoRight(),
state.isCanGoDown(),
state.isCanGoLeft(),
state.isFoodUp(),
state.isFoodUpRight(),
state.isFoodRight(),
state.isFoodDownRight(),
state.isFoodDown(),
state.isFoodDownLeft(),
state.isFoodLeft(),
state.isFoodUpLeft()
}
}));
}
My question is if the fit method is receiving correct parameters. If yes then my problem must be somwhere else.
In addition reward is calculated in a way that every time when snake steps in direction of the food its rewarded with 1, otherwise it's punished with -1.5. If it eats the food it's rewarded with 30.
Any hints and suggestions are welcome.
Related
I'm currently running into a rather annoying problem.
Im running round about this task, with a Spigot-1.8 in Java 1.8. But this exact code gives me two diffrent Results.
In one the Levelbar of Minecraft just counts down and in another the Levelbar just flickers every time it sets the new XP Value.
private static int buildTask;
private static int buildSeconds = 120;
public static void buildingTime() {
buildTask = Bukkit.getScheduler().scheduleSyncRepeatingTask(Plugin.getInstance(), () -> {
for (Player onlinePlayer : Bukkit.getOnlinePlayers()) {
onlinePlayer.setLevel(buildSeconds);
}
//executing other Actions with Actionbar and Broadcasting Seconds to all players
if (buildSeconds <= 0) {
//ending here
}
buildSeconds--;
}, 0, 20);
}
I'm using this Spigot: https://getbukkit.org/get/hNiHm0tuqAg1Xg7w7zudk63uHr0xo48D
Already solved.... Let two tasks run at the same time which conflicted
I have a game template which has two classes as its levels.
The game has a player, enemies, treasure chests, and walls.
When I run the program, it loads Level 1, successfully. I have designed the code so if a certain condition is met, Level 2 must load. Now, Level 2 loads. But certain elements from Level 1 remain on screen, ignoring certain elements from Level 2.
To be exact:
From Level 2, it loads the player's initial coordinates, the treasure chests, and the final destination, ignoring the enemies and the walls.
It brings with itself from Level 1 to Level 2: the walls, the treasure chests, and the enemies. The latter two are in a frozen state. They don't function.
//Instance variables:
private int lvlCount = 1;
// Game() method
public Game(Stage stage) {
this.score = new SimpleIntegerProperty(0);
this.mobs = new ArrayList<>();
this.hitBoxes = new ArrayList<>();
this.treasure = new ArrayList<>();
this.newLevel = new ArrayList<>();
this.enemyCount = 0;
bg();
loadLevel(1);
living();
stuff();
mob2();
finalText();
stage.setOnShown(e -> this.run());
}
//The switch for loading the Levels:
void loadLevel(int in) {
switch (in) {
case 1:
this.level = new Level1();
break;
case 2:
this.level = new Level2(); // placeholder for second level
break;
}
hitBoxes.clear(); // clear the list of hitboxes
mobs.clear(); // remove any existing old mobs
this.getChildren().addAll(hitBoxes); // add all the hitboxes for the walls to the scene
background.setImage(level.getImage()); // get the background image
hitBoxes.addAll(level.getWalls()); // get all the wall hitboxes
enemyCount = level.getEnemyCount(); // get the enemy count from the level
chestCount = level.treasureCount(); // get the treasure count from the level
this.initTreasure(chestCount); // add the treasure chests for the level
this.initMobs(enemyCount); // initialize our mobs
lvlCount = level.lvlCount();
this.initLevel(lvlCount);
}
//The collision check:
private void collisionCheck() {if (r.getFill().equals(Color.WHITE)){lvlCount--;}
//Main Game loop method:
private void play() {
AnimationTimer gameLoop = new AnimationTimer() {
public void handle(long arg0) {
for (int i = 0; i < mobs.size(); i++) {
MOB m = mobs.get(i);
if (m instanceof Enemy) {
((Enemy) m).moveCheck(arg0);
}
}
collisionCheck();
if (lvlCount == 0){
loadLevel(2);} //The condition that loads Level 2
gameLoop.start();
}
Please help me to figure out how I can load Level 2, successfully.
Please note that it does not produce any errors in the console. So, I am having a very hard time figuring out where the problem lies.
I only pasted the code that I thought is necessary to achieve my goal. Here you can see the full code, for veiwing purposes only, no download necessary:
http://paste.mooc.fi/955c6d30
Thank you very much in advance.
Here's my snippet of code. I have Lists for the blocks and player. All I need is for after the five seconds in that runnable is up, it'll replace the blocks that were previously replaced, assigning a player to get the blocks.
#EventHandler
public void onSnowballHit(ProjectileHitEvent e) {
// If it's a snowball...
if (e.getEntity() instanceof Snowball) {
Snowball snowball = (Snowball) e.getEntity();
final Player p = (Player) snowball.getShooter();
// ...if a player threw it...
if (snowball.getShooter() instanceof Player) {
// Make a Player from the Entity
BlockIterator iterator = new BlockIterator(e.getEntity().getWorld(),
e.getEntity().getLocation().toVector(), e.getEntity().getVelocity().normalize(),
0.0D, 4);
// Make a block
Block hitBlock = null;
// Loop through possible blocks
while (iterator.hasNext()) {
// Set the hitBlock to the current block we're checking
hitBlock = iterator.next();
// If it's not air, STOP!
if (!hitBlock.getType().equals(Material.AIR)) {
break;
}
}
int min = 1;
int max = 15;
Random r = new Random();
byte clayBlocks = (byte) (r.nextInt(max - min + 1) + min);
paintBlockList.add(hitBlock);
painters.add(p);
// Set it to stained clay
hitBlock.setType(Material.STAINED_CLAY);
// red = 14, blue = 11 (data values)
hitBlock.setData(clayBlocks);
Bukkit.getServer().getScheduler().scheduleSyncDelayedTask(this, new Runnable() {
public void run(){
if(painters.contains(p)){
painters.remove(p);
}
}
}, 100);
}
}
}
You could do it the same way you remove the Player from the painters list. Once you've found the first solid block (hitBlock) create a new final reference so that you can access it in the Runnable, for example like this:
final Block solidBlock = hitBlock;
To return the block to the state it was before you changed its type and data, you could keep track of those attributes:
final byte previousData = solidBlock.getData();
final Material previousType = solidBlock.getType();
Then in your run() method you can simply change the block back if it is still at that point different like so:
if (solidBlock.getType() != previousType) {
solidBlock.setType(previousType);
}
if (solidBlock.getData() != previousType) {
solidBlock.setData(previousData);
}
I'm sure there's a cleaner, nicer way to do it but this might be good enough for your purposes (I did find a glitch where if you throw two snowballs at the same block, it won't revert to the true original block but to a stained clay block because of the way the future Runnable tasks are written here, to fix this you'd need to write this completely differently and keep track of more things).
I try to improve the movement of my figures but i dont find the real reason why they stutter a bit. I am moving them with an SequenceAction containing an MoveToAction and an RunnableAction that does reset the moveDone flag so there can be started a new move.
The game itself is gridbased so if a move is done the squence starts a move to the next grid depending on the direction. So here is how it looks like:
note that this is inside of the Act of the figure
....//some more here
if (checkNextMove(Status.LEFT)) //check if the position is valid
{
status = Status.LEFT; //change enum status
move(Status.LEFT); // calls the move
screen.map.mapArray[(int) mapPos.x][(int) mapPos.y] = Config.EMPTYPOSITION;
screen.map.mapArray[(int) (mapPos.x - 1)][(int) mapPos.y] = Config.CHARSTATE;
mapPos.x--;
moveDone = false;
}
//... same for the up down right and so on.
//at the end of this checking the updating of the actor:
// methode from the absctract to change sprites
updateSprite(delta);
super.act(delta); // so the actions work
//end of act
And here is the move Method that does add the Actions
protected void move(Status direction)
{
// delete all old actions if there are some left.
clearActions();
moveAction.setDuration(speed);
//restart all actions to they can used again
sequence.restart();
switch (direction)
{
case LEFT:
moveAction.setPosition(getX() - Config.TILE_SIZE, getY());
addAction(sequence);
break;
case RIGHT:
moveAction.setPosition(getX() + Config.TILE_SIZE, getY());
addAction(sequence);
break;
case UP:
moveAction.setPosition(getX(), getY() + Config.TILE_SIZE);
addAction(sequence);
break;
case DOWN:
moveAction.setPosition(getX(), getY() - Config.TILE_SIZE);
addAction(sequence);
break;
default:
break;
}
}
The figures dont really move smothy.
Anyone does see a misstake or isnt it possible to let them move smothy like this?
It always stutter a bit if a new move is started. So i think this might not work good. Is there a differnt approach to move them exactly from one Grid to an nother? (Tried it myself with movementspeed * delta time but this does not work exactly and i struggeled around and used the Actionmodel)
it seems to make troubles with the one Frame where it does not move for example.
Here is an mp4 Video of the stuttering:
stuttering.mp4
just to mention, the camera movement is just fine. it's smothy but the figure stutters as you can see i hope
If you use your moveAction each move, you should call moveAction.restart() to reset the counter inside it:
// delete all old actions if there are some left.
clearActions();
sequence.reset(); // clear sequence
// add movementspeed. Note it can change!
moveAction.restart();
moveAction.setDuration(speed);
UPDATE:
Now issue occurs, because you call restart of SequenceAction after clearActions(). If your clearActions() removes all actions from SequenceAction then restart will be called for empty SequenceAction. So, do this instead:
//restart all actions to they can used again
sequence.restart();
// delete all old actions if there are some left.
clearActions();
moveAction.setDuration(speed);
Okay so i solved it myself. It has nothing todo with changing sequences around. It has something todo with the updatefrequency of the act() of the figures. The MoveToAction interpolates between the 2 points given by time. So if the last update, updates the MoveToAction by "to much time" it would need to go over the 100% of the action. So it would move to far but the action does not do this it set the final position and thats what it should do. Thats why it does not look fluent.
So how to solve this issue?
By decreasing the updatetime the "to far movement" decreases too because the steplength is getting smaller. So i need to increase the updatespeed above the 60fps. Luckily i got an thread for my updating of the figures and positions and so on. GameLogicThread. This ran on 60fps too. So i solved the stuttering by increasing it's frequence. up to around 210fps at the moment so the "overstep" is minimal. You can't even notice it.
To Show how my thread works:
public class GameLogicThread extends Thread
{
private GameScreen m_screen;
private boolean m_runing;
private long m_timeBegin;
private long m_timeDiff;
private long m_sleepTime;
private final static float FRAMERATE = 210f;
public GameLogicThread(GameScreen screen)
{
m_screen = screen;
setName("GameLogic");
}
#Override
public void run()
{
m_runing = true;
Logger.log("Started");
while (m_runing)
{
m_timeBegin = TimeUtils.millis();
// hanlde events
m_screen.m_main.handler.processEvents();
synchronized (m_screen.figureStage)
{
// now figures
if (m_screen.m_status == GameStatus.GAME)
{
m_screen.character.myAct(1f / GameLogicThread.FRAMERATE);// and here it is ;)
m_screen.figureStage.act(1f / GameLogicThread.FRAMERATE);
}
}
m_timeDiff = TimeUtils.millis() - m_timeBegin;
m_sleepTime = (long) (1f / GameLogicThread.FRAMERATE * 1000f - m_timeDiff);
if (m_sleepTime > 0)
{
try
{
Thread.sleep(m_sleepTime);
}
catch (InterruptedException e)
{
Logger.error("Couldn't sleep " + e.getStackTrace());
}
}
else
{
Logger.error("we are to slow! " + m_sleepTime);
}
}
}
public void stopThread()
{
m_runing = false;
boolean retry = true;
while (retry)
{
try
{
this.join();
retry = false;
}
catch (Exception e)
{
Logger.error(e.getMessage());
}
}
}
}
I am using this method in AndEngine to determine the scene being touched by the user.
#Override
public boolean onSceneTouchEvent(Scene pScene, TouchEvent pSceneTouchEvent) {
if(pSceneTouchEvent.isActionDown()) {
Log.e("Arcade", "Scene Tapped");
//Simulate player jumping
}
return false;
}
What i want to do is when the scene is tapped, i want to allow the player to jump.
Now two things for this would it be better to use PathModifier, or MoveYModifier considering it is landscape mode?
If either please provide an example of such.
Thanks
EDIT:
ive managed to use Physics to simulate a jump using this..
#Override
public boolean onSceneTouchEvent(Scene pScene, TouchEvent pSceneTouchEvent) {
if(pSceneTouchEvent.isActionDown()) {
Log.e("Arcade", "Scene Tapped");
final Vector2 velocity = Vector2Pool.obtain(mPhysicsWorld.getGravity().x * -0.5f,mPhysicsWorld.getGravity().y * -0.5f);
body.setLinearVelocity(velocity);
Vector2Pool.recycle(velocity);
return true;
}
return false;
}
As you said in the answer by changing the gravity. The only issue is, when the user keeps touching the screen the sprites keep going up and up and up. How can i set it where the user can only click once and cant make him jump again until the sprite hits the ground, which is a rectangle?
Use the MoveYModifier. remember, you can register as many modifiers as you want. So if, for example, the game a platform game and the character always moves on the X axis, and he can jumpt if he wants (Like Gravity Guy or Yoo Ninja, although these games change the gravity which is something else).
You could do like:
Entity playerEntity = ..//It doesn't matter if the player is a sprite, animated sprite, or anything else. So I'll just use Entity here, but you can declare your player as you wish.
final float jumpDuration = 2;
final float startX = playerEntity.getX();
final float jumpHeight = 100;
final MoveYModifier moveUpModifier = new MoveYModifier(jumpDuration / 2, startX, startX + jumpHeight);
final MoveYModifier moveDownModifier = new MoveYModifier(jumpDuration / 2, startX + jumpHeight, startX);
final SequenceEntityModifier modifier = new SequenceEntityModifier(moveUpModifier, moveDownModifier);
playerEntity.registerEntityModifier(modifier);
EDIT:
For your second question:
Create a variable boolean mIsJumping in your scene; When the jump starts - set it to true. If the user taps the screen and mIsJumping == true, don't jump.
Now, register a ContactListener to your PhysicsWorld. Whenever there is contact between the player and the ground, set mIsJumping to false.
There are many samples of using ContactListeners in AndEngine forums, a quick search yields some. If you need an example, you can ask for one :)
EDIT 2: ContactListener sample:
Have 2 variables to hold IDs for the player and the ground: private static final String PLAYER_ID = "player", GROUND_ID = "ground";
When you create the ground body and the player body, set their IDs as the user data: playerBody.setUserData(PLAYER_ID); and groundBody.setUserData(GROUND_ID);
Create a ContactListener as a field in your scene:
private ContactListener mContactListener = new ContactListener() {
/**
* Called when two fixtures begin to touch.
*/
public void beginContact (Contact contact) {
final Body bodyA = contact.getFixtureA().getBody();
final Body bodyB = contact.getFixtureB().getBody();
if(bodyA.getUserData().equals(PLAYER_ID)) {
if(bodyB.getUserData().equals(GROUND_ID))
mIsJumping = false;
}
else if(bodyA.getUserData().equals(GROUND_ID)) {
if(bodyB.getUserData().equals(PLAYER_ID))
mIsJumping = false;
}
}
/**
* Called when two fixtures cease to touch.
*/
public void endContact (Contact contact) { }
/**
* This is called after a contact is updated.
*/
public void preSolve(Contact pContact) { }
/**
* This lets you inspect a contact after the solver is finished.
*/
public void postSolve(Contact pContact) { }
};
Lastly, register that contact listener: physicsWorld.setContactListener(mContactListener);
EDIT 3:
To move your sprite over the X axis, you can apply a force using Body.applyForce method, or apply an impulse using Body.applyLinearImpulse method. Play around with the arguments and find what works the next.
The vector should consist a X part only; Try Vector2 force = Vector2Pool.obtain(50, 0);. Then apply the force this way: body.applyForce(force, body.getWorldCenter());.