Read data saved by spark redis using Java - java

I using spark-redis to save Dataset to Redis.
Then I read this data by using Spring data redis:
This object I save to redis:
#Getter
#Setter
#AllArgsConstructor
#NoArgsConstructor
#Builder
#RedisHash("collaborative_filtering")
public class RatingResult implements Serializable {
private static final long serialVersionUID = 8755574422193819444L;
#Id
private String id;
#Indexed
private int user;
#Indexed
private String product;
private double productN;
private double rating;
private float prediction;
public static RatingResult convert(Row row) {
int user = row.getAs("user");
String product = row.getAs("product");
double productN = row.getAs("productN");
double rating = row.getAs("rating");
float prediction = row.getAs("prediction");
String id = user + product;
return RatingResult.builder().id(id).user(user).product(product).productN(productN).rating(rating)
.prediction(prediction).build();
}
}
Save object by using spark-redis:
JavaRDD<RatingResult> result = ...
...
sparkSession.createDataFrame(result, RatingResult.class).write().format("org.apache.spark.sql.redis")
.option("table", "collaborative_filtering").mode(SaveMode.Overwrite).save();
Repository:
#Repository
public interface RatingResultRepository extends JpaRepository<RatingResult, String> {
}
I can't read this data have been saved in Redis by using Spring data redis because structure data saved by spark-redis and spring data redis not same (I checked value of keys created by spark-redis and spring data redis are different by using command: redis-cli -p 6379 keys \* and redis-cli hgetall $key)
So how to read this data have been saved using Java or by any library in Java?

The following works for me.
Writing data from spark-redis.
I use Scala here, but it's essentially the same as you do in Java. The only thing I changed is I added a .option("key.column", "id") to specify the hash id.
val ratingResult = new RatingResult("1", 1, "product1", 2.0, 3.0, 4)
val result: JavaRDD[RatingResult] = spark.sparkContext.parallelize(Seq(ratingResult)).toJavaRDD()
spark
.createDataFrame(result, classOf[RatingResult])
.write
.format("org.apache.spark.sql.redis")
.option("key.column", "id")
.option("table", "collaborative_filtering")
.mode(SaveMode.Overwrite)
.save()
In spring-data-redis I have the following:
#Getter
#Setter
#AllArgsConstructor
#NoArgsConstructor
#Builder
#RedisHash("collaborative_filtering")
public class RatingResult implements Serializable {
private static final long serialVersionUID = 8755574422193819444L;
#Id
private String id;
#Indexed
private int user;
#Indexed
private String product;
private double productN;
private double rating;
private float prediction;
#Override
public String toString() {
return "RatingResult{" +
"id='" + id + '\'' +
", user=" + user +
", product='" + product + '\'' +
", productN=" + productN +
", rating=" + rating +
", prediction=" + prediction +
'}';
}
}
I use CrudRepository instead of JPA:
#Repository
public interface RatingResultRepository extends CrudRepository<RatingResult, String> {
}
Querying:
RatingResult found = ratingResultRepository.findById("1").get();
System.out.println("found = " + found);
The output:
found = RatingResult{id='null', user=1, product='product1', productN=2.0, rating=3.0, prediction=4.0}
You may notice that the id field was not populated because the spark-redis stored has a hash id and not as a hash attribute.

Related

Getting entityManageFactory error. Schema-validation: missing table [hibernate_sequence]

I have 2 databases. One is set up and and it works. After I add second db I am having following error entityManageFactory error. Schema-validation: missing table [hibernate_sequence].
My db schema looks like this: db schema screenshot
I have two classes for two tables:
#Data
#AllArgsConstructor
#NoArgsConstructor
#Builder
#Entity(name = "nightly_rate_amounts")
#Table(name = "nightly_rate_amounts")
public class BookedNightlyRate {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
#Column(name = "bnr_meta_id")
private Long id;
#Column(name = "unit_uuid")
private UUID unitUuid;
private LocalDate firstLiveDate;
private LocalDate date;
private BigDecimal amount;
#Column(name = "currency_code")
private String currencyCode;
public ImmutableTriple<UUID, LocalDate, String> toUnitDateCurrencyKey() {
return new ImmutableTriple<>(unitUuid, date, currencyCode);
}
public ImmutablePair<UUID, String> toUnitCurrencyKey() {
return new ImmutablePair<>(unitUuid, currencyCode);
}
}
and:
#Data
#AllArgsConstructor
#NoArgsConstructor
#Builder
#Entity(name = "unit_attributes")
#Table(name = "unit_attributes")
public class BookedUnitAttributes {
#Id
#Column(name = "unit_uuid")
private UUID unitUuid;
#Column(name = "first_date_available")
private LocalDate firstLiveDate;
}
and Repository files:
public interface BookedNightlyRatesDao extends CrudRepository<BookedNightlyRate, Long> {
#Query(value = "SELECT DISTINCT bnr.unit_uuid as unitUuid, bnr.date, bnr.amount, bnr.currency_code as currencyCode " +
"FROM nightly_rate_amounts AS bnr " +
"WHERE bnr.unit_uuid IN (<unitUuids>) AND (bnr.date BETWEEN :fromDate AND :toDate)", nativeQuery = true)
List<BookedNightlyRate> findBookedNightlyRates(#Param("unitUuids") List<String> unitUuids, #Param("fromDate") LocalDate fromDate, #Param("toDate") LocalDate toDate);
#Query(value = "SELECT DISTINCT opb.unit_uuid as unitUuid, opb.date, opb.amount, opb.currency_code as currencyCode " +
"FROM opb_nightly_rate_amounts AS opb " +
"JOIN opb_sync_enabled_for_unit AS sync ON opb.unit_uuid = sync.unit_uuid WHERE sync.enabled = 1 AND opb.is_active = 1 " +
"AND sync.unit_uuid IN (<unitUuids>) AND (opb.date BETWEEN :fromDate AND :toDate)", nativeQuery = true)
List<BookedNightlyRate> findOPBRates(#Param("unitUuids") List<String> unitUuids, #Param("fromDate") LocalDate fromDate, #Param("toDate") LocalDate toDate);
}
second interface:
public interface BookedUnitAttributesDao extends CrudRepository<BookedUnitAttributes, UUID> {
#Query(value = "SELECT ua.unit_uuid as unitUuid, ua.first_date_available as firstLiveDate " +
"FROM unit_attributes AS ua " +
"WHERE ua.unit_uuid IN (<unitUuids>)", nativeQuery = true)
List<BookedUnitAttributes> findUnitAttributes(#Param("unitUuids") List<String> unitUuids);
}
I am rewriting my db from jdbi to jpa. So Data classes didn't have any annotations and I refactored my model files regarding it queries in repository files.
Since you add two database Spring dosn't know what kind of database it connect. You have to exactly showed what kind of database you want to connect.
You might confiugure connection with two different database here is example of working with JdbcTemplate connection.
#Configuration
#ComponentScan("uz.dbo.dbocallcenter")
#PropertySource("classpath:database.properties")
public class Config2 {
#Autowired
Environment environment;
private final String DRIVER = "driver";
private final String URL1 = "url1";
private final String USER1 = "dbusername1";
private final String PASSWORD1 = "dbpassword1";
private final String URL2 = "url2";
private final String USER2 = "dbusername2";
private final String PASSWORD2 = "dbpassword2";
private DataSource dataSource1() {
return getDataSource(URL1, USER1, PASSWORD1);
}
private DataSource dataSource2() {
return getDataSource(URL2, USER2, PASSWORD2);
}
private DataSource getDataSource(String url1, String user1, String password1) {
DriverManagerDataSource driverManagerDataSource = new DriverManagerDataSource();
driverManagerDataSource.setUrl(environment.getProperty(url1));
driverManagerDataSource.setUsername(environment.getProperty(user1));
driverManagerDataSource.setPassword(environment.getProperty(password1));
driverManagerDataSource.setDriverClassName(environment.getProperty(DRIVER));
return driverManagerDataSource;
}
#Bean(name = "jdbcTemplate2")
public JdbcTemplate jdbcTemplate2() {
return new JdbcTemplate(dataSource2());
}
#Bean(name = "jdbcTemplate1")
public JdbcTemplate jdbcTemplate1() {
return new JdbcTemplate(dataSource1());
}
}
you have to do with JpaRepository connection. More precisely you can gain knowledge about this source
https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#reference
Here is also good explanation how to connect two differnet database in one spring boot project
https://www.baeldung.com/spring-data-jpa-multiple-databases

o.h.engine.jdbc.spi.SqlExceptionHelper: ERROR: Column cliententi0_.name does not exist

Repostory
#Repository
public interface ClientRepository extends JpaRepository<ClientEntity, Long> {
#Modifying
#Transactional
#Query(value = "SELECT pp.id, TO_CHAR(pp.created_dt::date, 'dd.mm.yyyy')\n" +
"AS 'Data', CAST(pp.created_dt AS time(0)) AS 'Time', au.username AS 'UserName',\n" +
"ss.name AS 'Service', pp.amount AS 'Amount',\n" +
"REPLACE(pp.status, 'SUCCESS', 'Success') AS 'Payment_status', pp.account AS 'Account',\n" +
"pp.external_id AS 'Idn', COALESCE(pp.external_status, null, 'DN')\n" +
"AS 'Stat'\n" +
"FROM payments AS pp\n" +
"INNER JOIN user AS au ON au.id = pp.creator_id\n" +
"INNER JOIN services AS ss ON ss.id = pp.service_id\n" +
"WHERE pp.created_dt >= '2021-09-28'\n" +
"AND ss.name = 'Faberlic' AND pp.status = 'SUCCESS'", nativeQuery = true)
List<Client> getAllByRegDate();
}
Inteface
public interface Client {
Long getId();
#JsonFormat(shape = JsonFormat.Shape.STRING)
LocalDate getCreated_dt();
String getUsername();
String getName();
int getAmount();
String getStatus();
String getAccount();
String getExternal_id();
String getExternal_status();
}
DTO
#Getter
#Setter
#NoArgsConstructor
#AllArgsConstructor
#ToString
public class ClientDto {
private Long id;
#JsonFormat(shape = JsonFormat.Shape.STRING)
private LocalDate created_dt;
private String username;
private String name;
private int amount;
private String status;
private String account;
private String external_id;
private String external_status;
public ClientDto(Client client) {
this.id = client.getId();
/...
/...
this.external_status = client.getExternal_status();
}
public ClientDto(ClientDto clientDto) {
this.id = clientDto.getId();
/...
this.external_status = clientDto.getExternal_status();
}
public ClientDto(ClientEntity clientEntity) {
}
#Override
public String toString() {
return "" + id + "|" + created_dt + "|" + username + "|" + name +
"|" + amount + "|" + status + "|" + account + "|" + external_id + "|" + external_status;
}
}
Entity
#Getter
#NoArgsConstructor
#AllArgsConstructor
#Immutable
#Entity
#Table(name = "payments", schema = "public")
public class ClientEntity {
#Id
private Long id;
#Column(name = "created_dt")
private LocalDate created_dt;
#Column(name = "username")
private String username;
#Column(name = "name")
private String name;
#Column(name = "amount")
private int amount;
#Column(name = "status")
private String status;
#Column(name = "account")
private String account;
#Column(name = "external_id")
private String external_id;
#Column(name = "external_status")
private String external_status;
}
I am trying to save data to a csv file. I take data from one database, from three tables. In entity #Table in "name" I specify one of the existing tables - "payment". All data is taken from three tables (as I have written in Query). But when program is run, an error appears that the "name" column does not exist. This column is in another table from which I am fetching data. Can't figure out what I should do.
This is more of an answer to this question and the question you asked here, combined. Imho you are making things overly complex with your structure of having a Client interface which is used as a projection, which is then turned into a ClientDto (why? the projection is already a DTO) and you have your entities.
Instead of doing this just use a JdbcTemplate with a RowCallbackHandler to write the rows to CSV. This will use a lot less memory, be faster (as you aren't creating multiple objects per row to then throw it away, and you don't have all the rows in memory).
import java.io.FileWriter;
import java.sql.ResultSet;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
#Component
public class SchedulerService {
private static final String QUERY = "SELECT pp.id, pp.created_dt au.username, ss.name, pp.amount\n" +
"REPLACE(pp.status, 'SUCCESS', 'Success'), pp.account,\n" +
"pp.external_id AS 'Idn', COALESCE(pp.external_status, null, 'DN') AS 'Stat'\n" +
"FROM payments AS pp\n" +
"INNER JOIN user AS au ON au.id = pp.creator_id\n" +
"INNER JOIN services AS ss ON ss.id = pp.service_id\n" +
"WHERE pp.created_dt >= '2021-09-28'\n" +
"AND ss.name = 'Faberlic' AND pp.status = 'SUCCESS'";
private static final DateTimeFormatter date_format = DateTimeFormatter.ofPattern("dd.MM.yyyy");
private static final DateTimeFormatter time_format = DateTimeFormatter.ofPattern("HH:mm:ss");
private final JdbcTemplate jdbc;
public SchedulerService(JdbcTemplate jdbc) {
this.jdbc = jdbc;
}
#Scheduled(fixedRate = 5000)
public void downloadBlockedClients() {
String filename = "select.csv";
try (FileWriter writer = new FileWriter(filename)) {
writer.append("id|date|time|username|name|amount|status|account|external_id|external_status").append('\n');
this.jdbc.query(QUERY, (ResultSet rs) -> writeLine(writer, rs));
} catch (Exception e) {
e.printStackTrace();
}
}
private void writeLine(FileWriter writer, ResultSet rs) {
try {
LocalDateTime ldt = rs.getTimestamp("created_dt").toLocalDateTime();
writer.append(String.valueOf(rs.getLong("id")));
writer.append('|');
writer.append(ldt.format(date_format));
writer.append('|');
writer.append(ldt.format(time_format));
writer.append('|');
writer.append(rs.getString("username"));
writer.append('|');
writer.append(rs.getString("name"));
writer.append('|');
writer.append(String.valueOf(rs.getBigDecimal("amount")));
writer.append('|');
writer.append(rs.getString("status"));
writer.append('|');
writer.append(rs.getString("account"));
writer.append('|');
writer.append(rs.getString("idn"));
writer.append('|');
writer.append(rs.getString("stat"));
writer.append('\n');
} catch (Exception e) {
throw new IllegalStateException(e);
}
}
}
Something along these lines will make your resources more efficient (saves the copying, having results duplicated in memory) and should be faster. You could move the row handling to a method so your lambda gets a bit more readable.
NOTE: I assumed that you are using Spring Boot and that the `JdbcTemplate is available out-of-the-box. If not you need to configure one next to your JPA configuration.
#Bean
public JdbcTemplate jdbcTemplate(DataSource dataSource) {
return new JdbcTemplate(dataSource);
}

how to change object to string(java spring boot)

when i run the code i got the object
[[Ljava.lang.Object;#8f17f7c, [Ljava.lang.Object;#6c0a4f24,
[Ljava.lang.Object;#be4886c, [Ljava.lang.Object;#1760591d,
[Ljava.lang.Object;#14e9ce12, [Ljava.lang.Object;#2aa4c0c4,
[Ljava.lang.Object;#5ac9a14]
so.. i want to get the String result which in the below plz teach me the way
[Dataset_info(Ds_id=1111, ds_code=a, ds_name=e, ds_category=g, ds_stru=q, insert_ddtt=null, update_ddtt=null), Dataset_info(Ds_id=11111, ds_code=z, ds_name=eww, ds_category=g, ds_stru=q, insert_ddtt=null, update_ddtt=null)]
#Data
#Entity
#Table(name = "category")
#DynamicInsert
#DynamicUpdate
#NoArgsConstructor
#AllArgsConstructor
public class Category {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "category_id", columnDefinition = "INT(11)")
private Integer Category_id;
#Column(name = "name", columnDefinition = "VARCHAR(20)")
private String name;
#Column(name = "parent", columnDefinition = "int(11)")
private Integer parent;
}
this is my Category Code
#RestController
#RequestMapping(value = "/Category")
#Slf4j
public class CategoryController {
#Autowired CategoryRepository categoryRepository;
#RequestMapping(value = "/all", method =
RequestMethod.GET)
#ResponseBody
public String getCategoryList() {
List < Object[] > all =
this.categoryRepository.findByCategory();
return all.toString();
//log.info(query);
//return "Test";
}
}
this is my CategoryController code
import java.util.List;
#Repository
public interface CategoryRepository extends
JpaRepository < Category, Integer > {
public static final String FIND_PROJECTS = "SELECT t1.name
AS lev1,
t2.name as lev2,
t3.name as lev3,
t4.name as lev4
FROM category AS t1 LEFT JOIN category AS t2 ON t2.parent =
t1.category_id LEFT JOIN category AS t3 ON t3.parent =
t2.category_id LEFT JOIN category AS t4 ON t4.parent =
t3.category_id WHERE t1.name = 'ROOT'
";
#Query(value = FIND_PROJECTS, nativeQuery = true)
public List < Object[] > findByCategory();
}
this is my CategoryRepository Code
private void mysql2() {
this.categoryRepository.findByCategory();
}
this is my application Code for running
so plz teach me i crave to know the way
thank you
You can use Projection to contain these values :
public interface CategoryProjection {
public String getLev1();
public String getLev2();
public String getLev3();
public String getLev4();
}
Then use the interface Projection with Repository :
//...
#Query(value = FIND_PROJECTS, nativeQuery = true)
public List<CategoryProjection> findByCategory();
How to access values in Projections
Because It's a interface, only have getter method.
Using Loop (foreach loop, fori loop, ...)
ex :
List<CategoryProjection> list = categoryRepository.findByCategory();
list.forEach(c -> {
System.out.println(c.getLev1() + " - " + c.getLev2());
});
// loop i
for (int i = 0; i < list.size(); i++) {
System.out.println(list.get(i).getLev1() + " - " + list.get(i).getLev2());
}
Using index
ex : to get Object Category in index 0
String lev1 = list.get(0).getLev1();
UPD: In your case, I think you can change the method return type to List, HttpMessageConverter would convert the result as JSON String to client. Hope it help.
#ResponseBody
public List getCategoryList() {
List<Object[]> all = this.categoryRepository.findByCategory();
return all;
}
Override toString() method in your POJO object. For example:
public class Category {
private Integer Category_id;
private String name;
private Integer parent;
//omitted getter/setter
#Override
public String toString() {
return "Category{" +
"Category_id=" + Category_id +
", name='" + name + '\'' +
", parent=" + parent +
'}';
}
}

How to provide values from a class which is not an entity to a repository in spring data

I have a doubt about Spring Data and Spring Repositories.
I need to provide some values to a CrudRepository from another class which is not an Entity. For example:
I Have a class
#Entity
class Profile {
private String id;
private String name;
private long birthDate;
private String aboutMe;
...
}
and
class MyProfile {
private String profileId;
private String accountId;
private String name;
private String aboutMe;
}
and a Repository
#Transactional
#EnableTransactionManagement
#Repository
public interface ProfileRepository extends CrudRepository<Profile, String>{
#Transactional
Profile findByAccountId(String id);
void updateMyProfile(MyProfile myProfile);
}
and I would like to update only some fields from Profile using data provided in MyProfile. There is a way to do this?
Thanks!!
You will have to write the update query manually:
#Query("UPDATE Profile p" +
" SET " +
" p.name = ?#{#myprofile.name}, " +
" p.aboutMe = ?#{#myprofile.aboutMe}" +
" p.account.id = ?#{#myprofile.accountId}" + // I assume that account is another entity
" WHERE p.id = ?#{#myprofile.profileId}")
#Transactional
#Modifying
void updateMyProfile(#Param("myprofile") MyProfile myprofile);

Neo4J, Spring Data. How to query Relationship entity?

I use Neo4J database with Spring Data. I am unable to query (with custom query) a relationship directly to my Relation entity which looks like that:
#RelationshipEntity(type = "OCCURS_WITH")
public class Relation {
#GraphId
private Long id;
#StartNode
#Fetch
private Hashtag from;
#EndNode
#Fetch
private Hashtag to;
#GraphProperty(propertyType = long.class)
private Long[] timestamps = new Long[0];
private boolean active;
// getters, setters
}
I have also a repository interface as follow:
public interface RelationRepository extends CRUDRepository<Relation> {
#Query(value = " MATCH (h1)-[rel]->(h2) " +
" WHERE h1.name = {0} AND h2.name = {1}" +
" RETURN rel")
Relation find(String from, String to);
}
But when I query the repository I get an empty Relation object.
Everything works well when I am quering to dummy object in that way:
#Query(value = " MATCH (h1)-[r]->(h2) " +
" WHERE h1.name = {0} AND h2.name = {1} " +
" RETURN id(r) AS id, h1.name AS from, h2.name AS to, length(r.timestamps) AS size")
RelationshipData findData(String from, String to);
#QueryResult
public interface RelationshipData {
#ResultColumn("id")
String getId();
#ResultColumn("from")
String getFrom();
#ResultColumn("to")
String getTo();
#ResultColumn("size")
int getSize();
}
Is it possible to query directly to my entity?

Categories

Resources