Keywords

1 Introduction

In recent years, smartphone composing applications and music posting sites are well-supported [1,2,3]. As the environment in which composing activities can be easily done has become improved, novice musical composers and instrument users begin to occupy a part of users who use composition tools. In this paper, we propose a system to support novice musical composers.

For a novice musical composer or instrument user who is not easy to play, it is difficult “to produce a natural combination of melody and accompaniment” such as accompaniment production and the harmony of melody and accompaniment. Musical compositing support research thus far has realized a system to produce a combination of melody and accompaniment (hereinafter referred to as “music sketch”) by automatic musical composition such as automatic accompaniment generation [4] and automatic lead sheet generation [5]. However, for users such as who want to publish a produced song, because they have a main purpose of self-expression, it is desirable to be able to directly and easily input the composing elements of a melody such as a rhythm, tones and a chord. Furthermore, portability is also necessary to form their inspiration. Portable systems [2, 3] are also available, but their support is not enough compared to the existing composition support researches.

Therefore, in this paper, we propose a portable system that allows users to input a melody in real time during automatic playing of accompaniment, thereby enabling to directly and easily enter the composition elements of a melody. In this system, we use a smartphone to generate a corrected melody so as to be able to input a natural melody even on the small screen so that it is consistent with the accompaniment and the melody up to the time of input. It also outputs musical score and MIDI, enabling to coordinate with instrument playing and DTM. In this paper, in order to verify its usefulness, we released it to the general public as iOS and Android applications [6] to conduct extensive use experiments.

Fig. 1.
figure 1

Schematic chart of the proposed system.

Full size image
Fig. 2.
figure 2

Illustration of input screen.

Full size image
Fig. 3.
figure 3

Input screen of application.

Full size image

2 System Overview

We show the system outline in Fig. 1 to illustrate the flow of the proposed system following the numbers in the Figure.

The user produces a music sketch by (1) Selecting one out of 12 kinds of accompaniments prepared in advance, and (2) Inputting an melody extemporarily in real time during the automatic playing. The input screen at this time is shown in Figs. 2 and 3 and the select screen in shown in Fig. 4. As it is difficult to input it in real time in concert with the accompaniment at sight reading, display of a line showing the tempo(Tempo line and Time line), addition of percussion, and quantization of sound to reproduce were implemented as simplifying functions.

Fig. 4.
figure 4

Screen of selecting an accompaniment.

Full size image
Fig. 5.
figure 5

Screen of a music score.

Full size image

During the time of (2), since the system generates (3) pitches constituting a natural melody, input by the user becomes a natural melody. The pitch candidates to generate is multiplied by a key and constraints imposed by the last user’s input. Instead of arranging the pitch candidates randomly on the screen, in order to follow the relative pitch, the pitch candidates are arranged so that the higher the sound becomes as it advances in the positive direction of the horizontal axis of the screen, and the lower the sound becomes as it advances in the negative direction. Also, as long as the user continues to enter at the same position, the pitches are arranged from the position the user input last so that the same pitch is played.

Fig. 6.
figure 6

The algorithm of arranging pitches.

Full size image

This pitches arrangement can be defined by a recurrence formula starting from i as follows. The algorithm is shown in Fig. 6. Let \(S^{(k)}=\{s^{(k)}_1, s^{(k)}_2, ... , s^{(k)}_{|N|}\}\) be the pitch sequence on the input screen at the time of input number k and \(P^{(k)}=\{p^{(k)}_1, p^{(k)}_2, ... , p^{(k)}_{|M|}\}\) be the usable pitch candidate array. \(P^{(k+1)}\) is generated depending on the input pitch \(s^{(k)}_i\) and chord. At that time, the pitch sequence \(S^{(k+1)}\) at the time of input number \(k+1\) becomes as follows.

$$\begin{aligned} s^{k+1}_j = \left\{ \begin{array}{ll} s^{k}_i &{} (i = j) \\ \max \nolimits _{m, s^{(k+1)}_{j+1}>p^{(k+1)}_m}{p^{(k+1)}_m} &{} (i > j) \\ \min \nolimits _{m, s^{(k+1)}_{j-1}<p^{(k+1)}_m}{p^{(k+1)}_m} &{} (i < j). \end{array} \right. \end{aligned}$$
(1)

After inputting the melody, the system edits the input melody, and (4) outputs the music sketch in the form of music score or MIDI. The score screen is shown in Fig. 5.

Table 1. Questions for evaluation.
Full size table

3 Experiment

We implemented the system as a smartphone application “RoughON” [6] to release it to the general public. As of January 27, 2017, the number of unique users of the application reached 4300. In addition, the cumulative number of music sketch productions exceeds 10,000 (music: 9800, chord: 3500), and the cumulative output number in MIDI format reaches 3300. Furthermore, even now it is used about 100 times every day.

Fig. 7.
figure 7

Graph of experimental results.(Q3, Q5 show the average and SE. )

Full size image

A questionnaire was used for the evaluation experiment. We show the questionnaire, the each method and the number of subjects in Table 1. The user who installed the application and made music sketch more than 5 times was the subject, and the questionnaire was divided into 2 times according to the period. The number of subjects at the first time was 46 people, and the number of subjects at the second time was 38 people. However, because of questionnaire by public disclosure, the number of answers is different for each question item.

Question contents are classified into two types: question asking the attributes of subjects and question asking system evaluations obtained by subjects using “RoughON”. The question method selectively uses the choices (Q1, 2, 4, 6) and the points (Q3, 5), and furthermore a free answer column was provided. In the scoring system, we made a 5-point evaluation with 1 step (5: very good evaluation, 1: very bad evaluation). The results of the evaluation experiment are shown in Fig. 7.

In Q1 and 2, we asked the subjects’ experience in musical composition and instrument. In question Q1 asking the experience in musical composition, those who have no experience in musical composition and beginners accounted for about 80%. For Q4 asking the difficulty of music composition by musical instruments, about 40% answered that musical composition using musical instruments was possible, but nearly all of the respondents felt difficulty in musical composition using musical instruments.

Q3 and Q4 are questions concerning the evaluation as a system for supporting the music composition. Q3 asked the support of initial music composing, which is the purpose of the proposed system, having gotten 3.57 with a 5-point score evaluation. Compared to the score evaluation, in the free description column, many favorable opinions were obtained, saying that “the operation was intuitive and easy.” Besides, there are also answers: “we can compose a music with a game feeling,” “it is difficult to compose a music from the beginning, but we can easily take the first step if we use RoughON.” When Q4 asked which one would you like to use: RoughON or other composition applications?, and 47 people out of 57 people who had use experience of other music composition applications answered that they want to use RoughON. As a reason for that, there were many answers: “simple and easy to use compared to other applications,” similarly in Q3. In addition, there was also an answer that “Although other applications inhibit the pleasure of making a music because of a full automatic system, RoughON reasonably supports the composer.” Besides, favorable answers for the accompaniment function, such as “it is easier to do if accompaniment is available at first” and “it can superimpose a sound easily” were obtained. In Q3 and 4, it was shown that the proposed system is useful for supporting the early stage of music composition.

Q5 and 6 are questions about the sufficiency of music sketch to develop it into a music. In the score evaluation conducted in Q5 asking whether music sketch would become a clue to complete a music, the average value was 4.0. Also, in Q6 asking if you would like to actually use music sketch, there were 23 subjects who answered that they would use it. They answered various uses, such as “they make use of it to their own song” as a place to use and “they use for the BGM of radio, game and movie.” Q5 and 6 showed that the music sketch produced by the proposed system plays a role of helping to complete a music.

4 Conclusion

In this paper, we proposed a portable system for music composition and instrument novices to produce a music sketch with real time input. In addition, the proposed system was implemented, released to the general public, and evaluated as a smartphone application: “RoughON.” As a result, it was shown that the proposed system played a role of supporting the first step of music composition, revealing that its output, music sketch, is suitable as an intermediate product for completing a music.

A wide range of experiments were carried out by public questionnaire, so many subjects were able to be obtained. However, problems such as not being able to confirm the detailed operation of the examinee, not being able to confirm whether the subject used the music sketch produced by the proposed system or not occurred. In the future, we will acquire operation history and plan to investigate more detailedly for users who use the proposed system frequently.