Hey i try to do the first steps with Casssandra trigger. The trigger should only get the mutation and write it to a simple .txt file nothing more nothing less.
Everytime i do an isert i get the following Error:
java.lang.AbstractMethodError: org.apache.cassandra.triggers.invertedindex.augment(Lorg/apache/cassandra/db/partitions/Partition;)Ljava/util/Collection
The code is from a example i found in the internet.
public class invertedindex implements ITrigger
{
// private static final Logger logger = LoggerFactory.getLogger(invertedindex.class);
private Properties properties = loadProperties();
//private Object array;
public void augment(ByteBuffer key, ColumnFamily update) throws IOException
{
PrintWriter pWriter = null;
//long millis2;
// List<RowMutation> mutations = new ArrayList<>();
String indexKeySpace = properties.getProperty("keyspace");
String indexColumnFamily = properties.getProperty("table");
for (IColumn cell : update)
{
// Skip the row marker and other empty values, since they lead to an empty key.
if (cell.value().remaining() > 0)
{
pWriter = new PrintWriter(new BufferedWriter(new FileWriter("log_trigger_test.txt",true)));
RowMutation mutation = new RowMutation(indexKeySpace, cell.value());
// mutation.add(properties.getProperty("columnfamily"), cell.name(), cell.value(), System.currentTimeMillis();
// mutations.add(mutation);
// mutation.addColumnOrSuperColumn(arg0, arg1);
//System.out.println((mutation.toString()));
pWriter.println((mutation.toString()));
}
}
pWriter.close();
// return mutations;
}
private static Properties loadProperties()
{
Properties properties = new Properties();
InputStream stream = invertedindex.class.getClassLoader().getResourceAsStream("invertedindex.properties");
try
{
properties.load(stream);
}
catch (Exception e)
{
throw new RuntimeException(e);
}
finally
{
FileUtils.closeQuietly(stream);
}
return properties;
}
}
What i am doing wrong here? And is there more information about Casssandra trigger anywere? I only found some old stuff?
It looks like you are using Cassandra 3.x but have written a trigger for pre-3.x. Your trigger should be implementing the:
public Collection<Mutation> augment(Partition update);
method.
Take a look at the trigger example here for how to implement a 3.x trigger.
Related
I want to use the tumbling window function for my program (non keyed data) as it is processing streaming data but only 300 messages/sec. I want to take it to at least 5K/sec. For this purpose, I want to use the tumbling window for 2 sec just to see speed up its performance. But I am not sure how to use this in my case.
Note: I am using the Geomesa HBase platform for saving the messages.
Also, I did not paste my whole application code here as I only need the window function for which this code is sufficient here for your understanding
Here is my flink code
public class Tranport {
public static void main(String[] args) throws Exception {
// fetch runtime arguments
String bootstrapServers = "xx.xx.xxx.xxx:xxxx";
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// Set up the Consumer and create a datastream from this source
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", bootstrapServers);
properties.setProperty("group.id", "group_id");
final FlinkKafkaConsumer<String> flinkConsumer = new FlinkKafkaConsumer<>("lc", new SimpleStringSchema(), properties);
flinkConsumer.setStartFromTimestamp(Long.parseLong("0"));
DataStream<String> readingStream = env.addSource(flinkConsumer);
readingStream.rebalance().map(new RichMapFunction<String, String>() {
private static final long serialVersionUID = -2547861355L; // random number
DataStore lc_live = null;
SimpleFeatureType sft_live;
SimpleFeatureBuilder SFbuilderLive; // feature builder for live
List<SimpleFeature> lc_live_features; //
#Override
public void open(Configuration parameters) throws Exception {
System.out.println("In open method.");
// --- GEOMESA, GEOTOOLS APPROACH ---//
// define connection parameters to xxx GeoMesa-HBase DataStore
Map<String, Serializable> params_live = new HashMap<>();
params_live.put("xxxx", "xxx"); // HBase table name
params_live.put("xxxx","xxxx");
try {
lc_live = DataStoreFinder.getDataStore(params_live);
if (lc_live == null) {
System.out.println("Could not connect to live");
} else {
System.out.println("Successfully connected to live");
}
} catch (IOException e) {
e.printStackTrace();
}
// create simple feature type for x table in HBASE
StringBuilder attributes1 = new StringBuilder();
attributes1.append("xxx:String,");
attributes1.append("xxx:Long,");
attributes1.append("source:String,");
attributes1.append("xxx:String,");
attributes1.append("xxx:Double,");
attributes1.append("status:String,");
attributes1.append("forecast:Double,");
attributes1.append("carsCount:Integer,");
attributes1.append("*xxx:Point:srid=4326");
sft_history = SimpleFeatureTypes.createType("xxxx", attributes1.toString());
try {
lc_history.createSchema(sft_history);
} catch (IOException e) {
e.printStackTrace();
}
// Initialize the variables
numberOfMessagesProcessed = 0;
numberOfMessagesFailed = 0;
numberOfMessagesSkipped = 0;
// for lc_Live
lc_live_features = new ArrayList<>();
SFbuilderLive = new SimpleFeatureBuilder(sft_live);
Here I want to create a Tumbling window function (Window All) which can take all the
stream messages with in 2 seconds of window and push them into the array list which i have created below
// live GeoMesa-HBase DataStore
// copy the list into a local variable and empty the list for the next iteration
List<SimpleFeature> LocalFeatures = live_features;
live_features = new ArrayList<>();
LocalFeatures = Collections.unmodifiableList(LocalFeatures);
try (FeatureWriter<SimpleFeatureType, SimpleFeature> writer = live.getFeatureWriterAppend(sft_live.getTypeName(), Transaction.AUTO_COMMIT)) {
System.out.println("Writing " + LocalFeatures.size() + " features to live");
for (SimpleFeature feature : LocalFeatures) {
SimpleFeature toWrite = writer.next();
toWrite.setAttributes(feature.getAttributes());
((FeatureIdImpl) toWrite.getIdentifier()).setID(feature.getID());
toWrite.getUserData().put(Hints.USE_PROVIDED_FID, Boolean.TRUE);
toWrite.getUserData().putAll(feature.getUserData());
writer.write();
}
} catch (IOException e) {
e.printStackTrace();
}
It's late but might help someone. In Scala, you can do something like
env.addSource(consumer).
windowAll(TumblingProcessingTimeWindows.of(Time.seconds(2)))
But, remember if you are not using KeyBy(), then your data won't be processed in parallel no matter what value you set in env.setParallelism()
I'm trying to implement a wrapped "move" function with Xodus, but something is not working out right:
#Override
public boolean move(String appId, String name, String targetName) {
final boolean[] success = new boolean[1];
final Environment env = manager.getEnvironment(xodusRoot, appId);
final VirtualFileSystem vfs = manager.getVirtualFileSystem(env);
env.executeInTransaction(
new TransactionalExecutable() {
#Override
public void execute(#NotNull final Transaction txn) {
File file = vfs.openFile(txn, name, false);
InputStream input = vfs.readFile(txn, file);
if(input != null) {
File targetFile = vfs.openFile(txn, targetName, true);
DataOutputStream output = new DataOutputStream(vfs.writeFile(txn, targetFile));
try {
output.write(ByteStreams.toByteArray(input));
} catch (IOException e) {
e.printStackTrace();
}
vfs.deleteFile(txn, name);
success[0] = true;
}
}
});
// vfs.shutdown();
// env.close();
return success[0];
}
The problem is the file gets moved but the byte array is not getting copied, not sure if the problem is because of multiple VFS operation in the same transaction. Can someone give me a hint of why the bytes from the source file are not getting copied properly?
Looks like you are trying to implement another version of VirtualFileSystem.renameFile(..).
I'm creating a little java app and I'm trying to load the yml files based on config.yml lang set (en/it) but I can't find a way to load them, only the last one in an array is loaded which is "it" for me.
I know that my method is probably the worst solution for a language file, I'm open to every method that will help me with the problem. But I prefer an external lang_en/it file instead of internal ones (Or is it better internal?)
After I set the language, the app will self-update every text in every class.
static final Properties props = new Properties();
static WelcomeMessage main = new WelcomeMessage();
static File file = null;
static File folder = null;
static boolean os = main.os.startsWith("Windows");
public static void create() {
String[] lang = {"en", "it"};
for (String s : lang) {
file = new File(WelcomeMessage.user + "/AppData/Roaming/MyApp/lang_" + s + ".yml");
folder = new File(file.getParent());
SetLanguages(s);
}
if (!file.exists()) {
try {
if (os) {
folder.mkdir();
file.createNewFile();
} else {
file = new File(main.user + "/Library/Application Support/MyApp/config.yml");
folder.mkdir();
file.createNewFile();
}
} catch (Exception e) {
System.out.println(e + " " + file);
}
}
}
public static void SetLanguages(String lang) {
if (lang.equals("en")) {
store("Settings.Save", "Save");
store("Settings.ConfigPath", "Config Path");
store("Settings.Language", "Language");
store("Settings.Title", "Settings");
} else if (lang.equals("it")) {
store("Settings.Save", "Salva");
store("Settings.ConfigPath", "Percorso config");
store("Settings.Language", "Lingua");
store("Settings.Title", "Impostazioni");
}
}
public static String get(String value) {
String key = null;
try {
FileInputStream in = new FileInputStream(file);
props.load(in);
key = props.getProperty(value);
in.close();
} catch (Exception fnf) {
System.out.println(fnf);
}
return key;
}
public static void store(String value, String key) {
try {
FileOutputStream out = new FileOutputStream(file);
props.setProperty(value, key);
props.store(out, null);
out.close();
} catch (Exception fnf) {
System.out.println(fnf);
}
}
This is how I get a text from yml:
path.setText(Language.get("Settings.ConfigPath"));
language.setText(Language.get("Settings.Language"));
f.setTitle(Language.get("Settings.Title"));
save.setText(Language.get("Settings.Save"));
And this my Language.get(key)
public static String get(String value) {
String key = null;
try {
FileInputStream in = new FileInputStream(file);
props.load(in);
key = props.getProperty(value);
in.close();
} catch (Exception fnf) {
System.out.println(fnf);
}
return key;
}
I suggest the following changes:
Create a Settings class to hold the properties save, configPath, language and title. Even better if this class uses an immutable builder pattern, because once set, the properties will never change.
Create a SettingsFactory class with method getSettings(language). This class shall also have a field Map<String, Settings>. In the constructor (or a static block), first check if a file exists on the disk, and if yes, load it into the map. If not, populate the map, one entry for each language, and persist to the disk.
getSettings would simply return the value from the map corresponding to the given language.
The format of the file written to the disk is a different matter. You say YAML, but I'm not seeing any YAML specific code in your snippet. If you don't know how to write a map to YAML, open a different question.
Every now and then I get the exception pointed out in the title. Does anyone know what exception 4 means or where this is documented? I've looked at questions relating to this and I've learned that there's exception 2 which apparently means "No such file" and exception 3 which is "Permission Denied". However, when I get nested exception is 4, there is no description so I'm not sure what this means. Is it just a generic read time out? This happens very infrequently, and it's difficult to reproduce, as some of our users have no issues at all, while some run into this issue every now and then.
org.springframework.core.NestedIOException: failed to read file /ftadv:D=IBM-037,C=UTF-8/__'MAIN.FRAME.DATASET'; nested exception is 4:
at org.springframework.integration.sftp.session.SftpSession.readRaw(SftpSession.java:143)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.readRaw(CachingSessionFactory.java:268)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.readRaw(CachingSessionFactory.java:268)
at com.my.package.FTPService.ftpStoreAuthData(FTPService.java:166)
at com.my.package.FTPService.ftpStoreNetAuthData(FTPService.java:59)
at com.my.package.FTPEndpoint.ftpStoreNetAuthData(FTPEndpoint.java:27)
Here's my config if it helps.
#Configuration
class SftpConfiguration {
#Inject
private Environment env;
SessionFactory<LsEntry> getSftpSessionFactory() {
DefaultSftpSessionFactory sftpSessionFactory =
new DefaultSftpSessionFactory(Boolean.parseBoolean(env.getRequiredProperty("sftp.isSharedSession")));
sftpSessionFactory.setHost(env.getRequiredProperty("sftp.host"));
sftpSessionFactory.setPort(Integer.parseInt(env.getRequiredProperty("sftp.port")));
sftpSessionFactory.setUser(env.getRequiredProperty("sftp.id"));
sftpSessionFactory.setPrivateKey(new FileSystemResource(env.getRequiredProperty("sftp.keyPath")));
sftpSessionFactory.setPrivateKeyPassphrase(env.getRequiredProperty("sftp.passphrase"));
sftpSessionFactory.setTimeout(Integer.parseInt(env.getRequiredProperty("sftp.timeout")));
sftpSessionFactory.setAllowUnknownKeys(Boolean.parseBoolean(env.getRequiredProperty("sftp.allowUnknownKeys")));
return new CachingSessionFactory<LsEntry>(sftpSessionFactory);
}
CachingSessionFactory getCachingSessionFactory(){
CachingSessionFactory cachingSessionFactory = new CachingSessionFactory(getSftpSessionFactory());
cachingSessionFactory.setSessionWaitTimeout(Integer.parseInt(env.getRequiredProperty("sftp.sessionWaitTimeout")));
cachingSessionFactory.setPoolSize(Integer.parseInt(env.getRequiredProperty("sftp.poolSize")));
return cachingSessionFactory;
}
}
And here's an example of the calling code:
#Service
class FTPService {
private static final org.apache.logging.log4j.Logger logger = LogManager.getLogger(FTPEndpoint.class);
private CachingSessionFactory cachingSessionFactory;
private String adviceString;
#Inject
private FTPService(SftpConfiguration sftpConfig) {
this.cachingSessionFactory = sftpConfig.getCachingSessionFactory();
this.adviceString = sftpConfig.getAdviceString();
}
private String ftpStoreAuthData() {
Session session = this.cachingSessionFactory.getSession();
String mainframeDataSet = "'MAIN.FRAME.DATASET'";
try(BufferedInputStream inputStream = new BufferedInputStream(session.readRaw(adviceString + mainframeDataSet));
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream))
{
Stream<String> lines = reader.lines());
return doThingsWithFTPData(lines);
} catch (IOException ex) {
LogException.writeToFile(ex.getMessage(),ex.getStackTrace(), env.getProperty("sftp.exceptionsPath"));
}finally {
try {
if (session.isOpen()) {
session.finalizeRaw();
session.close();
}
}catch (IOException ioe){
System.out.println(ioe.getLocalizedMessage() + ioe.getCause().toString());
LogException.writeToFile(ioe.getMessage(),ioe.getStackTrace(), env.getProperty("sftp.exceptionsPath"));
throw new IOException();
}
}
return "test";
}
}
I have a static util class that does some string manipulation on a bit sensitive data.
Prior to use of this class I need to initialize certain static variables with values, such as usernames/password, that I prefer to store in a .properties file.
I am not very familiar with how loading of .properties file work in Java, especially outside of *Spring DI *container.
Anyone can give me a hand/insight on how this can be done?
Thank you!
Addition: .properties file precise location is unknown, but it will be on the classpath. Sorta like classpath:/my/folder/name/myproperties.propeties
First, obtain an InputStream from which the properties are to be loaded. This can come from a number of locations, including some of the most likely:
A FileInputStream, created with a file name that is hard-coded or specified via a system property. The name could be relative (to the current working directory of the Java process) or absolute.
A resource file (a file on the classpath), obtained through a call to getResourceAsStream on the Class (relative to the class file) or ClassLoader (relative to the root of the class path). Note that these methods return null if the resource is missing, instead of raising an exception.
A URL, which, like a file name, could be hard-coded or specified via a system property.
Then create a new Properties object, and pass the InputStream to its load() method. Be sure to close the stream, regardless of any exceptions.
In a class initializer, checked exceptions like IOException must be handled. An unchecked exception can be thrown, which will prevent the class from being initialized. That, in turn, will usually prevent your application from running at all. In many applications, it might be desirable to use default properties instead, or fallback to another source of configuration, such as prompting a use in an interactive context.
Altogether, it might look something like this:
private static final String NAME = "my.properties";
private static final Properties config;
static {
Properties fallback = new Properties();
fallback.put("key", "default");
config = new Properties(fallback);
URL res = MyClass.getResource(NAME);
if (res == null) throw new UncheckedIOException(new FileNotFoundException(NAME));
URI uri;
try { uri = res.toURI(); }
catch (URISyntaxException ex) { throw new IllegalArgumentException(ex); }
try (InputStream is = Files.newInputStream(Paths.get(uri))) { config.load(is); }
catch (IOException ex) { throw new UncheckedIOException("Failed to load resource", ex); }
}
Check out java.util.Properties.
You can use a static initializer. So on the top of the class you can do:
static {
Properties props = new Properties();
InputStream steam = ...; // open the file
props.load(stream);
// process properties content
String username = props.getProperty("username");
}
Use either:
CurrentClassName.class.getResourceAsStream
new FileInputStream(File)
to get the input stream depending on if the class is in or out of the classpath. Then use
Properties.load
to load the properties.
It's been a while, but if I remember correctly you just do something like this:
Properties prop = new Properties();
prop.load(new FileInputStream(filename));
//For each property you need.
blah = prop.getProperty(propertyname);
Well with static Properties it would make sense to initialize them as a Singleton which will be loaded once in a class. Here's an example:
class Example
{
public final static String PROPSFILE = "test.properties";
private static Properties props;
protected static Properties getProperties()
{
if(props == null)
{
props = new Properties();
props.load(new FileInputStream(new File(PROPSFILE));
}
return props;
}
public static User getUser()
{
String username = getProperties().getProperty("username");
return new User(username);
}
}
If you use relative Pathnames you should make sure, that your classpath is setup righ.
for me MyClass.class.getClassLoader().getResourceAsStream(..) did the trick:
private static final Properties properties;
static {
Properties fallback = new Properties();
fallback.put(PROP_KEY, FALLBACK_VALUE);
properties = new Properties(fallback);
try {
try (InputStream stream = MyClass.class.getClassLoader().getResourceAsStream("myProperties.properties")) {
properties.load(stream);
}
} catch (IOException ex) {
// handle error
}
}
I agree with #Daff, maybe better to use singleton class...this what i have on my project for similar requirement, maybe it may help:
clients of the class can use it like this:
ConfigsLoader configsLoader = ConfigsLoader.getInstance("etc/configs.xml");
System.out.format("source dir %s %n", configsLoader.getSourceDir());
and then the class:
public class ConfigsLoader {
private String sourceDir;
private String destination;
private String activeMqUrl;
private static Logger log = Logger.getLogger(ConfigsLoader.class.getName());
private static ConfigsLoader instance = null;
private ConfigsLoader(String configFileName) {
log.info("loading configs");
Properties configs = new Properties();
try {
configs.loadFromXML(new FileInputStream(configFileName));
sourceDir = configs.getProperty("source.dir");
destination = configs.getProperty("destination");
activeMqUrl = configs.getProperty("activemqconnectionurl");
configs.setProperty("lastLoaded", new SimpleDateFormat("yyyy-M-d HH:mm").format(new Date()));
configs.storeToXML(new FileOutputStream(configFileName), "saving last modified dates");
} catch (InvalidPropertiesFormatException e) {
log.log(Level.SEVERE,"Error occured loading the properties file" ,e);
} catch (FileNotFoundException e) {
log.log(Level.SEVERE,"Error occured loading the properties file" ,e);
} catch (IOException e) {
log.log(Level.SEVERE,"Error occured loading the properties file" ,e);
}
}
public static ConfigsLoader getInstance(String configFileName) {
if(instance ==null) {
instance = new ConfigsLoader(configFileName);
}
return instance;
}
public String getSourceDir() {
return sourceDir;
}
public void setSourceDir(String sourceDir) {
this.sourceDir = sourceDir;
}
public String getDestination() {
return destination;
}
public void setDestination(String destination) {
this.destination = destination;
}
public String getActiveMqUrl() {
return activeMqUrl;
}
public void setActiveMqUrl(String activeMqUrl) {
this.activeMqUrl = activeMqUrl;
}
}
I did this finally using getResourceAsStream() fuction associated with the class in which the static code block is being written.
//associate Property and ImputStream imports
public class A {
static Properties p;
static {
p = new Properties();
try {
InputStream in = A.class.getResourceAsStream("filename.properties");
p.load(in);
} catch (FileNotFoundException e) {
System.out.println("FileNotFoundException");
e.printStackTrace();
} catch (IOException e) {
System.out.println("IOException");
e.printStackTrace();
}
}
.
.
.
}